All of Holden Karnofsky's Comments + Replies

Future-proof ethics

Hm. I contacted Nick and replaced it with another link - does that work?

Yup, works for me now.
Comments for shorter Cold Takes pieces

I didn't make a claim that constant replacement occurs "empirically." As far as I can tell, it's not possible to empirically test whether it does or not. I think we are left deciding whether we choose to think of ourselves as being constantly replaced, or not - either choice won't contradict any empirical observations. My post was pointing out that if one does choose to think of things that way, a lot of other paradoxes seem to go away.

Comments for shorter Cold Takes pieces

I personally like Radiohead a lot, but I don't feel like my subjective opinions are generally important here; with Pet Sounds I tried to focus on what seemed like an unusually clear-cut case (not that the album has nothing interesting going on, but that it's an odd choice for #1 of all time, especially in light of coming out a year after A Love Supreme).

Comments for shorter Cold Takes pieces

I think this is interesting and plausible, but I'm somewhat skeptical in light of the fact that there doesn't seem to have been much (or at least, very effective) outcry over the rollback of net neutrality.

The Wicked Problem Experience

I think this is often a good approach!

3Yonatan Cale3mo
This is the kindest way anyone ever told me that I didn't help ;) <3 <3 If anyone's interested, I just posted about this idea yesterday: []
Important, actionable research questions for the most important century

I think "people can test their fit without much experience, but would get lots of value out of that experience for actually doing this work" is pretty valid, though I'll also comment that I think there are diminishing returns to direct experience - I think getting some experience (or at least exposure, e.g. via conversation with insiders) is important, but I don't think one necessarily needs several years inside key institutions in order to be helpful on problems like these.

Important, actionable research questions for the most important century

I don't have anything available for this offhand - I'd have to put serious thought into what questions are at the most productive intersection of "resolvable", "a good fit for Metaculus" and "capturing something important." Something about warning signs ("will an AI system steal at least $10 million?") could be good.

Consider trying the ELK contest (I am)

Thanks! I'd estimate another 10-15 hours on top of the above, so 20-30 hours total. A good amount of this felt like leisure time and could be done while not in front of a computer, which was nice. I didn't end up with "solutions" I'd be actually excited about for substantive progress on alignment, but I think I accomplished my goal of understanding the ELK writeup well enough to nitpick it.

Future-proof ethics

The link works for me in incognito mode (it is a Google Drive file).

Huh, maybe someone else wants to weigh in? When I view in an incognito window, it prompts me to login. When I view it logged in, it says "You need access. Ask for access, or switch to an account with access." I'm not sure if you are the owner, but if so, you likely just need to click on "Share", then "Restricted" in the Get Link dialog (it doesn't really look like you can click there, but you can), then change the setting to "Anyone with the link".
Future-proof ethics

Thanks, this is helpful! I wasn't aware of that usage of "moral quasi-realism."

Personally, I find the question of whether principles can be described as "true" unimportant, and don't have much of a take on it. My default take is that it's convenient to sometimes use "true" in this way, so I sometimes do, while being happy to taboo it anytime someone wants me to or I otherwise think it would be helpful to.


Future-proof ethics

I share a number of your intuitions as a starting point, but this dialogue (and previous ones) is intended to pose challenges to those intuitions. To follow up on those:

On Challenge 1A (and as a more general point) - if we take action against climate change, that presumably means making some sort of sacrifice today for the sake of future generations. Does your position imply that this is "simply better for some and worse for others, and not better or worse on the whole?" Does that imply that it is not particularly good or bad to take action on climate chan... (read more)

As you noticed, I limited the scope of the original comment to axiology (partly because moral theory is messier and more confusing to me), hence the handwaviness. Generally speaking, I trust my intuitions about axiology more than my intuitions about moral theory, because I feel like my intuition is more likely to "overfit" on more complicated and specific moral dilemmas than on more basic questions of value, or something in that vein. Anyway, I'll just preface the rest of this comment with this: I'm not very confident about all this and at any rate not sure whether deontology is the most plausible view. (I know that there are consequentialists who take person-affecting views too, but I haven't really read much about it. It seems weird to me because the view of value as tethered seems to resist aggregation, and it seems like you need to aggregate to evaluate and compare different consequences?) Since in deontology we can't compare two consequences and say which one is better, the answer depends on the action used to get there. I guess what matters is whether the action that brings about world X involves us doing or neglecting (or neither) the duties we have towards people in world X (and people alive now). Whether world X is good/bad for the population of world X (or for people alive today) only matters to the extent that it tells us something about our duties to those people. Example: Say we can do something about climate change either (1) by becoming benevolent dictators and implementing a carbon tax that way, or (2) by inventing a new travel simulation device, which reduces carbon emissions from flights but is also really addictive. (Assume the consequences of these two scenarios have equivalent expected utility, though I know the example is unfair since "dictatorship" sounds really bad -- I just couldn't think of a better one off the top of my head.) Here, I think the Kantian should reject (1) and permit or even recommend (2), roughly speaking because (2) resp
Future-proof ethics

I think that's a fair point. These positions just pretty much end up in the same place when it comes to valuing existential risk.

Future-proof ethics
That seems reasonable re: sentientism. I agree that there's no knockdown argument against lexicographic preferences, though I find them unappealing for reasons gestured at in this dialogue.
Future-proof ethics

It's interesting that you have that intuition! I don't share it, and I think the intuition somewhat implies some of the "You shouldn't leave your house" type things alluded to in the dialogue.

I'm pretty happy to bite that bullet, especially since I'm not an egoist. I should still leave my house because others are going to suffer far worse (in expectation) if I don't do something to help, at some risk to myself. It does seem strange to say that if I didn't have any altruistic obligations then I shouldn't take very small risks of horrible experiences. But I have the stronger intuition that those horrible experiences are horrible in a way that the nonexistence of nice experiences isn't. And that "I" don't get to override the preference to avoid such experiences, when the counterfactual is that the preferences for the nice experiences just don't exist in the first place.
Future-proof ethics

I agree with this argument for discount rates, but I think it is a practical rather than philosophical argument. That is, I don't think it undermines the idea that if we were to avert extinction, all of the future lives thereby enabled should be given "full weight."

Future-proof ethics

You're right that I haven't comprehensively addressed risk aversion in this piece. I've just tried to give an intuition for why the pro-risk-aversion intuition might be misleading.

Future-proof ethics
I appreciate Kenny's comments pointing toward potentially relevant literature, and agree that you could be a utilitarian without fully biting this bullet ... but as far as I can tell, attempts to do so have enough weird consequences of their own that I'd rather just bite the bullet. This dialogue gives some of the intuition for being skeptical of some things being infinitely more valuable than others.
Future-proof ethics
I think you lose a lot when you give up additivity, as discussed here and here.
I understand that you lose a lot (and I appreciate your blog posts). But that is not an argument that additivity is correct. As I've written for my upcoming book: Imagine a universe that has only two worlds, World R and World FL. In World R, Ricky the Rooster is the only sentient being, and is suffering in an absolutely miserable life. This is bad. But where is it bad? In Ricky’s consciousness. And nowhere else. On World FL, Rooster Foghorn is living in one forest and Rooster Leghorn is living in a separate forest. They are the World FL’s only sentient beings, and don’t know each other. Their lives are as bad as Ricky’s. Our natural response is to think that World FL is twice as bad as World R. But where could it possibly be twice as bad? Foghorn’s life is bad in his consciousness and nowhere else. Leghorn’s life is bad in his consciousness and nowhere else. Where is their world twice as bad as Ricky’s? Nowhere. Okay, yes, I admit it is twice as bad in your mind and my mind. But we are not part of that universe. Imagine that these worlds are unknown to any other sentient being. Then there is simply nowhere that World FL is worse than World R. In this universe, there are three worlds and only three worlds: one in each of their minds. Tell me where I am factually wrong. Please, I’m asking you. My life would be much easier and happier if you would. Don’t say that the implications of this insight leads to absurd conclusions that offend our intuitions. I already know that! Just tell me where am I factually wrong. I know (oh, yes, I know) that this seems like it can’t possibly be right. This is because we can’t help but be utilitarian in this regard, just like we can’t help but feel like we are in control of our consciousness and our decisions and our choices. But I can see no way around this simple fact: morally-relevant “badness” exists only in individual consciousnesses.
Future-proof ethics
I think I agree with Tyler. Also see this follow-up piece - "future-proof" is supposed to mean "would still look good if we made progress, whatever that is." This is largely supposed to be a somewhat moral-realism-agnostic operationalization of what it means for object-level arguments to be right.
Future-proof ethics
I don't think we should assume future ethics are better than ours, and that's not the intent of the term. I discuss what I was trying to do more here.
“Biological anchors” is about bounding, not pinpointing, AI timelines

Sorry for the long delay, I let a lot of comments to respond to pile up!

APS seems like a category of systems that includes some of the others you listed (“Advanced capability: they outperform the best humans on some set of tasks which when performed at advanced levels grant significant power in today’s world (tasks like scientific research, business/military/political strategy, engineering, and persuasion/manipulation) … “). I still don’t feel clear on what you have in mind here in terms of specific transformative capabilities. If we condition on not ha... (read more)

I'm a fan of lengthy asynchronous intellectual exchanges like this one, so no need to apologize for the delay. I hope you don't mind my delay either? As usual, no need to reply to this message. I think I agree with this. Re: quantification: I agree; currently I don't have good metrics to forecast on, much less good forecasts, for persuasion stuff and AI-PONR stuff. I am working on fixing that problem. :) Re persuasion: For the past two years I have agreed with the claims made in "The misinformation problem seems like misinformation."(!!!) The problem isn't lack of access to information; information is more available than it ever was before. Nor is the problem "fake news" or other falsehoods. (Most propaganda is true.) Being politically polarized and extremist correlates positively with being well-informed, not negatively! (Anecdotally, my grad school friends with the craziest/most-extreme/most-dangerous/least-epistemically-virtuous political beliefs were generally the people best informed about politics. Analogous to how 9/11 truthers will probably know a lot more about 9/11 than you or me.) This is indeed an epistemic golden age... for people who are able to resist the temptations of various filter bubbles and the propaganda of various ideologies. (And everyone thinks themself one such person, so everyone thinks this is an epistemic golden age for them.) I do disagree with your claim that this is currently an epistemic golden age. I think it's important to distinguish between ways in which it is and isn't. I mentioned above a way that it is. Agreed. I argued this, in fact. Disagree. I mean, I don't know, maybe this is true. But I feel like we shouldn't just throw our hands up in the air here, we haven't even tried! I've sketched an argument for why we should expect epistemic responsiveness to decrease in the near future (propaganda and censorship are bad for epistemic responsiveness & they are getting a lot cheaper and more effective & no pro-epistemic-respon
3Patrick Wilson2mo
Dear Holden and all Karnofskyites , Thanks for this great post and discussion - I really enjoyed the audio too. I began to compose a comment here but then it rambled on and on, and dived into various weird rabbit holes, and then I realised I needed to do more reading. I ended up writing a full-length essay over Easter and have just posted it on my new blog 'Path findings []'. I launched this a few weeks ago inspired by reading your post 'Learning by Writing' - and yay it seems that really works! Anyway, here's the post , fresh off the slab RABBITS, ROBOTS AND RESURRECTION [HTTPS://PATHFINDINGS.SUBSTACK.COM/P/RABBITS-ROBOTS-AND-RESURRECTION] RIFFING WITH KARNOFSKY ON THE VALUE OF PRESENT AND FUTURE LIVES, TO CELEBRATE THE 50TH ANNIVERSARIES OF 'WATERSHIP DOWN', 'LIMITS TO GROWTH' AND THE ALCOR FOUNDATION... I'd be thrilled if you could take a few moments to read or at least skim it, and would welcome any and all feedback, however brutal! Up front I confess not all the arguments are consistent, and the puns are consistently terrible, but I hope it makes some kind of sense. It will appeal particularly to people who like philosophy, ecology and rabbits, and features a lovely illustration Lyndsey Green. As a taster, here are some of the section headers (and most of the terrible puns): * Warren peace: a brief history of British rabbits * Too many bunnies? Malthus bites back * Abundant lives: valuing people now and in future * Staying alive: trolling the trolley problems * Of bunnies and bugs: who qualifies as people? * Back to life, back to reality… being human You have been warned! Best regards, Patrick
1Luís Campos3mo
FYI, the audio on the recording is slightly weird. :)
2Jacob Valero3mo
Thanks for this post! I found the inner dialogue very relatable and it was helpful in thinking about my own uncertainties.
I think the title of this post doesn't quite match the dialogue. Most of the dialogue is about whether additional good lives is at least somewhat good. But that's different from whether each additional good life is morally equivalent to a prevented death. The former seems more plausible than the latter, to me. Separating the two will lead to some situations where a life is bad to create but also good to save, once started. That seems more like a feature than a bug. If you ask people in surveys, my impression is that some small fraction of people say that they'd prefer to not have been born and that some larger fraction of people say that they'd not want to relive their life again — without this necessarily implying that they currently want to die.
The link to Chapter 2 of On the Overwhelming Importance of Shaping the Far Future at the end links to a non-public Google Drive file.
Really like this post! I think one important crux here is differing theories of value. My preferred theory is the (in my view, commonsensical) view that for something to be good or bad, it has to be good or bad for someone. (This is essentially Christine Korsgaard's argument; she calls it "tethered value".) That is, value is conditional on some valuer. So where a utilitarian might say that happiness/well-being/whatever is the good and that we therefore ought to maximise it, I say that the good is always dependent on some creature who values things. If all the creatures in the world valued totally different things than what they do in our dimension, then that would be the good instead. (I should mention that, though I'm not very confident about moral philosophy, to me the most plausible view is a version of Kantianism. Maybe I give 70% weight to that, 20% to some form of utilitarianism and the rest to Schopenhauerian ethics/norms/intuitions. I can recommend being a Kantian effective altruist: it keeps you on your toes. Anyway, I'm closer to non-utilitarian Holden in the post, but with some differences.) This view has two important implications: * It no longer makes sense to aggregate value. As Korsgaard puts it, "If Jack would get more pleasure from owning Jill's convertible than Jill does, the utilitarian thinks you should take the car away from Jill and give it to Jack. I don't think that makes things better for everyone. I think it makes it better for Jack and worse for Jill, and that's all. It doesn't make it better on the whole." * It no longer makes sense to talk about the value of potential people. Their non-existence is neither good nor bad because there is no one for it to be good or bad for. (Exception: They can still be valued by people who are alive. But let's ignore that.) I haven't spent tons of time thinking about how this shakes out in longtermism, so quite a lot of uncertainty here. But here's roughly how I think this
3Richard Y Chappell3mo
Great dialogue! As an additional 'further reading' suggestion, I just want to plug the 'Population Ethics []' chapter at It summarizes some less well-known possibilities (such as "value blur" in the context of a critical range view) that might avoid some of the problems of the (blur-free) total view.
A big difference in button 1 (small benefit for someone) and 1A (small chance of a small benefit for a large number of people) is the kind of system required for these outcomes. Button 1 requires basically a days worth of investment by someone making a choice to give it to another. Button 1A requires... perhaps a million times as much effort? We're talking about the equivalent of passing a national holiday act. This ends up requiring an enormous amount of coordination and investment. And the results do not scale linearly at all. That is, a person investing a day's worth of effort to try and pass a national holiday act don't have a 10E-8 chance of working. They have a much much smaller chance. Many many orders of magnitude less. In other words, the worlds posited by a realistic interpretation of what these buttons mean are completely different, and the world where button 1A process succeeds is to be preferred by at least six orders of magnitude. In other words, the colloquial understanding of the "big" impact is closer to right than the multiplication suggests. I'm not sure exactly how that impacts the overall conclusions, but I think this same dynamic applies to several odd conclusions -- the flaw is that the button is doing much much much more work in some situations than in others described as identical, and that descriptive flaw is pumping our intuitions to ignore those differences rather than address them.
I started writing a comment, then it got too long, so I put in my shortform here [] . :)
I don't necessarily disagree with your conclusion, but I don't know how you can feel sure about weighing a chicken's suffering vs a person. But I definitely disagree with the initial conclusion, and I think it is because you don't fear extreme suffering enough. If everyone behind the veil of ignorance knew what the worst suffering was, they would fear it more than they would value time at the beach. Re: longtermism, I find the argument in Pinker's latest book to be pretty compelling: The optimal rate at which the discount the future is a problem that we face not just as individuals but as societies, as we decide how much public wealth we should spend benefit our older selves and future generations. Discount it we must. It's not only that a current sacrifice would be in vain if an asteroid sends us the way of the dinosaurs. It's also that our ignorance of what the future will bring, including advances in technology, grows exponentially the farther out we plan. It would have made little sense for our ancestors a century ago to have scrimped for our benefit - say, diverting money from schools and roads to a stockpile of iron lungs to prepare for a polio epidemic - given that we're six times richer and have solved some of their problems while facing new ones they could not have dreamed of.
6Kenny Easwaran4mo
I don't think your argument against risk aversion fully addresses the issue. You give one argument for diversification that is based on diminishing marginal utilities, and then show that this plausibly doesn't apply in global charities. However, there's a separate argument for diversification that is actually about risk itself, and not diminishing marginal utility. You should look at Lara Buchak's book, "Risk and Rationality", which argues that there is a distinct form of rational risk-aversion (or risk-seeking-ness). On a risk neutral approach, each outcome counts in exact proportion to its probability, regardless of whether it's the best outcome, the worst, or in between. On a risk averse approach, the relative weight of the top ten percentiles of outcomes is less than the relative weight of the bottom ten percentiles of outcomes, and vice versa for risk seeking approaches. This turns out to precisely correspond to ways to make sense of some kinds of inequality aversion - making things better for a worse off person improves the world more than making things equally much better for a better off person. None of the arguments you give tell against this approach rather than the risk-neutral one. One important challenge to the risk-sensitive approach is that, if you make large numbers of uncorrelated decisions, then the law of large numbers kicks in and it ends up behaving just like risk neutral decision theory. But these cases of making a single large global-scale intervention are precisely the ones in which you aren't making a large number of uncorrelated decisions, and so considerations of risk sensitivity can become relevant.
Comments for shorter Cold Takes pieces

(Placeholder for comments on "To Match the Greats, Don’t Follow In Their Footsteps")

Comments for shorter Cold Takes pieces

Placeholder for comments on Beach Boys post

3Peli Grietzer23d
The obvious answer to what frame of mind are you missing here is that you have to actually like the genre of music Pet Sounds is working in relation to.
Simply put, there isn't any popular, jazz, or avant-garde music that was written and produced like Pet Sounds before Pet Sounds. It's literally an unprecedented work of art. It's not just about the fact that it had an exorbitant budget, but the fact that it was composed and directed almost singlehandedly by Wilson. It's the fact that Wilson found a way to be as successful and sophisticated to the degree that he was. He created commercial AM radio pop music with very complex forms and structures within an industry whose markets clamored for either simple three-chord rock 'n' roll, bubblegum, throwaway novelty songs, or schmaltz - an accomplishment that no one had thought possible back then. Every track contains dozens of different musical parts played and sung by a full-sized virtual orchestra (virtual because many of the parts - mostly the vocals - were overdubbed). These tracks were designed to be as intricate as possible without forcing the listener to struggle with the bombardment of information they're receiving. And it worked. Newly married couples around the world still choose God Only Knows as their wedding song - a song that was crafted so incredibly well that nobody notices that it has no key center until they try to learn to play it. "But it's doubtful that it used the recording studio better than today's music does." I don't know, I guess this comes down to whether you prefer: A) the kind of music that would be composed by one guy with strange ideas about music, recorded organically with analogue equipment and real singers and musicians, and then released as-is B) the kind of music that is composed by algorithms, programmed in a DAW, recorded with autotuned singers, and then screened by test audiences to take out all the "weird" parts "But it's inevitable that pop music would have gone in more complex directions." Another weird point. It's also inevitable that man will develop civilizations on other planets. Is the "inevitability" supposed make it a
1Bill Benzon4mo
To some extent I think a comparison between Pet Sounds and A Love Supreme is apples and qumquats. But still...I suspect that someone who is capable of listening to and understanding A Love Supreme, whether or not they like it, is also capable to listening to and understanding Pet Sounds, whether or not they like it. But I don't think the converse is necessarily true. That is, having the ability to listen to and understand Pet Sounds does not imply that one can also understand A Love Supreme or, for that matter, a Beethoven piano sonata.
1Simon 4mo
Thanks for this interesting series of blog posts! The podcast Strong Songs has a great episode [] about the song 'God Only Knows' that explains why the song/Pet Sounds is so special
1mother box5mo
i wondered whilst reading through this if framing / comparing your take on a “complex” album (like A Love Supreme) is useful if we don’t actually dive into why you think that is more complex and layered than an album like Pet Sounds. would it be useful to contextualise A Love Supreme within Coltrane’s works as well and across the jazz landscape of the time? would a jazz purist consider Coltrane’s album, which is arguably in the popular music arena (as much as a jazz album can be), to be less complex than other contemporaries that someone with a more intense affinity for jazz might pick out (making assumptions here about your love/knowledge of jazz, but also thinking of the people i know that can only reference A Love Supreme or Kind of Blue when talking about great jazz). and do we all suffer from comparing pop rock music to pop jazz music and giving overwhelming weight to jazz just because of its supposed higher end status (which leads me back to the Beethoven writings and the way people perceive older classical music vs new music, etc.). these were all the questions that popped in my head when reading and would love to find some deep dives into these things.
3JS Denain5mo
I notice that your playlist does not include Smile [], which Good Vibrations was meant for, and which I find more complex, impressive, and interesting than Pet Sounds. I wouldn't call it cohesive though, it's pretty all over the place. Perhaps rock critics give Pet Sounds credit for what Smile could have been, had it been completed? I'd be curious about Luke Muehlhauser's take on the questions you raise in the post, given his previous writing on similar subjects (e.g. on Scaruffi []). Finally, you cite Cowen a few times in the Beethoven post: I think you may find Paul McCartney as management study [] relevant.
I’d love to hear your take after doing a similar listen-through of Radiohead’s discography up through Kid A. They dominated the top 5 of Pitchfork’s reader poll of the best albums of all time, and that judgement feels much more representative to me of what a contemporary rock fan might choose than Pet Sounds.
I've felt flummoxed for a while about Pet Sounds. I first tried listening to it in high school (after learning of its acclaim) and couldn't make it through. When I listen to it now, over a decade later, I feel I can clearly hear and appreciate the "symphonic" quality of the songs, and the care and craft that went into the production, instrumentation, and compositions. It's not difficult for me to believe that it was a major leap forward and I think it's not too difficult to hear how influential it's been. A song I love, "John Allyn Smith Sails" by Okkervil River is partly an adaptation of "Sloop John B". Moreover, when I listen to Pet Sounds with 'audiophile brain' the sounds, melodies, and harmonies all sound great. But I just don't enjoy listening to the album. The vocals sound detached and clinical to me. For such an acclaimed and highly-ranked album, I feel it doesn't have many raw emotional hooks. Compare to others on the top of the lists Holden linked: Marvin Gaye, Nirvana, The Beatles, Joni Mitchell, Dylan. Their songs have some powerful emotional energy that Pet Sounds seems to lack-- and will typically make you feel something, even if it's not your cup of tea. To me, Pet Sounds sounds like the odd one out, so I still feel confused why it's so high on these lists. Also, I would definitely rank A Love Supreme much closer to the top.
Other than A Love Supreme, what albums have you found really impactful? Could you write 500-1000 words off the top of your head on why one of those albums is a work of genius? Have you ever taken over a conversation at a party (without really intending to) to explain how good some piece of music is and how everyone should go home and listen to it right away? I am really really not trying to call you a philistine, there is nothing wrong with not having super strong feelings about music. But my guess is that most music critics (professional or armchair) would answer yes to both. If you don't answer yes to these questions, maybe you're not responding to music the way that other people do. (Which, again, is okay.) Personally, fine art (painting, sculpture) does almost nothing for me. I couldn't offer any authentic opinion at all on whether Jackson Pollack was a more important artist than Georgia O'Keefe. So I have to assume that the people who do care about that question are perceiving something that I'm not. For what it's worth I consider Pet Sounds to be sublimely beautiful, but I have no idea how I'd explain what exactly is so beautful about it in a way that would convince anyone else.
I had a similar reaction on my first listen to Pet Sounds. I think the impact it had on production means that you need to have not heard any music after it to fully hear its importance. It sounds like a solid pop album to my ears. God only knows sounds beautiful to me but not other-worldly. But I'm assuming it would have blown my mind (as it did the Beatles) had I not heard the last 50 years of music.

I read your article and one element I think you might be missing, is the impact that Pet Sounds had on music production.

A Love Supreme is great, but it is pretty simple from a production standpoint. A group of talented musicians playing great music together.

Pet Sounds, on the other hand, is IMO widely regarded as an innovative musical production masterpiece. So leaving the quality of the songs aside, I recommend re-listening (maybe on high-end headphones) to how each of the sounds has been placed and fit together. I think often when people describe the alb... (read more)

Comments for shorter Cold Takes pieces

Comments for Cost disease and civilizational decline will go here.

1Ben Wōden5mo
You write "Is it in fact the case that the difference between the 1st- and 2nd-best performer should shrink as the number of competitors goes up? This isn't obvious to me either way." I think that, if you're drawing without replacement n times from a normal distribution, the difference between highest and second-highest value drawn should shrink as n rises, but that the opposite is true if the distribution is log-normal. I would expect that "greatness" in terms of critical acclaim in some field is log-distributed, so, the bigger the field, the greater the extent to which the leader should stand out above the second-best.
It is so, so much worse. I investigated the claim in depth, found it was indeed "ballistically false" (the claim being that the source they cite supports what the book says, not whether what they say in the book is actually true), and then decided to find out if Wengrow had perhaps apologized and issued a retraction . I ran across this Twitter thread: [] in which Wengrow defends his scholarship by variously claiming that people misread the source, that people misunderstood the claims in the source, that the source saying that "many" whites chose to stay with the Indians is evidence for his claim that they "almost invariably" did so, and that his source is unreliable and should not be trusted. This guy is either very stupid, a very bad liar, or just lazy and thinks that everyone else is a gullible idiot.
2JP Addison5mo
I believe Scott Alexander has cited this book’s “ballistically false” claim, and I definitely remember ~believing it and finding it strongly compelling.
Comments for shorter Cold Takes pieces

I largely agree with this comment, and I didn't mean to say that different intellectual property norms would create more "Beethoven-like" figures critical-acclaim-wise. I more meant to say it would just be very beneficial to consumers. (And I do think music is in a noticeably better state (w/r/t the ease of finding a lot that one really likes) than film or books, though this could be for a number of reasons.)

One reason film may be in a worse state could be that it takes many more people to make a film - one person's idea/vision almost always has to pass through many more filters. They cost more to make and there is more pressure to make it into something that will be widely successful to recoup those up front investments. Books I'm not so sure. It seems harder to write a novel to me, but maybe that's just because music comes more easily to me than writing. It strikes me that it's a much bigger time commitment to read enough of a novel to decide if you actually like it than it does to listen to a song and do the same. Perhaps this leads to self-publishing not being as viable option. Consumers rely more on filters/gatekeepers because you could spend a lifetime trying to sift through self-published novels and not find many good ones. Music may have the advantage of being able to be consumed somewhat passively - while driving, working, etc., while movies and books are a more immersive. More basically, you can consume astronomically more songs in a lifetime than books or movies.
Comments for shorter Cold Takes pieces

Sorry, just saw this! This did not in fact work out on the hoped-for timeline, and I didn't have a grantee in mind - I think the right way to try to do something here would've been through direct dialogue with policymakers.

Comments for shorter Cold Takes pieces

In response to the paragraph starting "I see how ..." (which I can't copy-paste easily due to the subscripts):

I think there are good pragmatic arguments for taking actions that effectively hold Ht responsible for the actions of Ht-1. For example, if Ht-1 committed premeditated murder, this gives some argument that Ht is more likely to harm others than the average person, and should be accordingly restricted for their benefit. And it's possible that the general practice of punishing Ht for Ht-1's actions would generally deter crime, while not creating other... (read more)

Rowing, Steering, Anchoring, Equity, Mutiny

I think this is a good point, but it doesn't totally knock me out of feeling sympathy for the "rowing" case.

It looks quite likely to me that factory farming is going to end up looking something like air pollution - something that got worse, then better, as capabilities/wealth improved. I expect the combination of improving "animal product alternatives" (Impossible, Beyond, eventually clean meat) with increasing wealth to lead this way.

Granted, this is no longer a "pure trend extrapolation," but I think the consistent and somewhat mysterious improvement in ... (read more)

“Biological anchors” is about bounding, not pinpointing, AI timelines

Interesting, thanks! Yep, those probabilities definitely seem too high to me :) How much would you shade them down for 5 years instead of 15? It seems like if your 5-year probabilities are anywhere near your 15-year probabilities, then the next 5 years have a lot of potential to update you one way or the other (e.g., if none of the "paths to PONR" you're describing work out in that time, that seems like it should be a significant update).

I'm not going to comment comprehensively on the paths you laid out, but a few things:

  • I think EfficientZero is sample-

... (read more)
Gaah, sorry, I keep forgetting to put links in -- APS-AI means Advanced, Planning, Strategically Aware AI -- the thing the Carlsmith report [] talks about. I'll edit to put links in retroactively. I've written a short story about what I expect the next 5 years to look like []. Insofar as AI progress is systematically slower and less impressive than what is depicted in that story, I'll update towards longer timelines, yeah. I'm currently at something like 20% that AI-PONR [] will be crossed in the next 5 years, and so insofar as that doesn't seem to have happened 5 years from now then that'll be a 20%-sized blow to my timelines in the usual Bayesian way. It's important to note that this won't necessarily lengthen my timelines all things considered, because what happens in those 5 years might be more than a 20% blow to 20+year timelines. (For example, and this is what I actually think is most likely, 5 years from now the world could look like it does at the end of my short story, in which case I'd have become more confident that the point of no return will come sometime between 2026 and 2036 than I am now, not less, because things would be more on track towards that outcome than they currently seem to be.) Re: persuasion tools: You seem to have a different model of how persuasion tools cause PONR than I do. What I have in mind is mundane, not exotic--I'm not imagining AIs building QAnon-like cult followings, I'm imagining the cost of censorship/propaganda* continuing to drop rapidly and the effectiveness continuing to increase rapidly, and (given a few years for society to catch up) ideological strife to intensify in general. This in turn isn't an x-risk by itself but it's certainly a risk factor, and insofar as our impact co
Comments for shorter Cold Takes pieces

I generally put this comment up in advance of the post, so that I can link to it from the post. The post is up now!

Checks out. Wasn't aware of that!
Recent post [] responding to you
3Rohin Shah5mo
Another potential reason that empowerment could lead to more onerous stakeholder management is that we're able to take more large-scale, impactful actions, and so it's much more common to have affected stakeholders than it was in the past.
Great post, I think this captures something very important about how the increasing size of, and focus on, externalities leads to more stakeholder vetos. I think you're correct that the underlying cause is individuals, but I do think there is something here about the solution. Private businesses have always had to deal with stakeholders, like suppliers and workers, and have historically been able to deal with this relatively well, because costs to these stakeholders could be compensated with fungible dollars. This allows for mutually beneficial agreements, competition and so on. In contrast, many of the stakeholder vetoes that are created by government do not allow such solutions: paying off stakeholders is considered bribery rather than legitimate payment. It's true that making rights alienable is compatible with a relatively high degree of government oversight, but most people would probably regard it as a move in a libertarian direction.
Did you take this post down? Or does it not exist yet?
Re: net neutrality, I have no insider knowledge, this is just my personal opinion as an observer. Little has changed since the NN repeal precisely because there was a relatively strong outcry at the time. It's hard to think of another issue that polls with 60-80% support across both parties. Practically, "little has changed" in the sense that AFAIK in these 4 years no ISP has switched to a business model based on charging internet companies for access to "fast lanes". IMO this is only because introducing tiered pricing would likely generate significant backlash, and ISPs have good reason to believe that, given the outcry at the time of repeal. The downsides of NN include unpredictable tail risks of a kind of lock-in that is very hard to undo. At the time of repeal, I think there were basically two categories of "sky is falling" rhetoric. (1) rational actors trying to drum up public opposition despite knowing that the worst-case scenario is unlikely, and (2) media actors who jumped on the NN bandwagon, simply because it generated engagement. Doesn't make sense for (1) to state "I was wrong" takes because nothing in these past 4 years falsifies the claim that eroding NN could gradually lead to an ossified internet with (much more) rent-seeking ISPs. (2) probably wouldn't recant anything since "we were wrong" stories seem like ineffective clickbait. In short, I think nothing bad has happened yet because people were so fired up about NN in the first place, and because practically a rent-seeking ISP would need more time to capitalize on the repeal.
Comments for shorter Cold Takes pieces

[Placeholder for Why it matters if "ideas are getting harder to find" comments]

For those particularly concerned with counterfactual impact, this is an argument to work on problems or in fields that are just beginning or don’t exist yet in which many of the wins haven’t been realized; this is not a novel argument. I think the bigger update is that “ideas get harder to find” indicates that you may not need to have Beethoven’s creativity or Newton’s math skills in order to make progress on hard problems which are relatively new or have received little attention. In particular, AI Safety seems like a key place where this rings true, in my opinion.
1Bill Benzon4mo
I vote for innovation as mining. I've visualized an abstract version of that starting on p. 14 ("Stagnation, Redux: Like diamonds, good ideas are not evenly distributed") of this working paper, What economic growth and statistical semantics tell us about the structure of the world [] , piggy-backing on Romer's 1992, Two Strategies for Economic Development.
Reading this post reminded me of someone whose work may be interesting to look into: Rufus Pollock, a former academic economist who founded the Open Knowledge Foundation []. His short book (freely available here []) makes the case for replacing traditional IP, like patents and copyright, with a novel kind of remuneration. The major benefits he mentions include increasing innovation and creativity in art, science, technology, etc.
I'll talk exclusively about music (mostly the broader rock/pop realm), because that’s the area that I know the best (being a lifelong obsessive music lover who has played in bands and dabbled in music production and DJing). It seems pretty clear to me that what you describe is already the current state in the world of music - and perhaps that’s partly why we don't see any more Beethovens? Riffing on past work is, arguably, something every musician does, consciously, or unconsciously. They cover songs, "steal" riffs, sample, and combine ideas from other artists. It's quite rare that you find someone even attempting to do something completely without precedent. (This seems so self-evident that I'm not going to provide examples, but I would be happy to, upon request.) Population growth/demographics, but also technology (recording and distribution, not even AI) have already resulted in exponential growth of the amount of music being produced. You used to have to pay to go into a studio to record an album, and get a record contract to distribute it. As a consumer, you'd have to work harder too. If you read about some music, you'd have to go to a store to find it and buy it, sometimes without every hearing it. Now people record in their bedrooms, upload to Bandcamp, Soundcloud, Spotify, etc., and consumers can find it online immediately. (Certainly, it seems like the evolution of things like the OpenAi Jukebox []will blow this up to absurd proportions.) Why has this not resulted in more universally acclaimed music? Perhaps partly because of intense competition? These days, many more musicians can earn some income from their music, but it's spread much more thin. Few can earn enough to make a living. This has resulted in a flourishing of many genres and sub-genres that appeal more narrowly, but may have a more consistent audience. When Beethoven was alive, you had his genius, but nowhere near the breadth and variety of music available tod
Consider trying the ELK contest (I am)

Hey Josh, I think this is a good point - it would be great to have some common knowledge of what sort of commitment this is.

Here's where I am so far:

  1. I read through the full report reasonably carefully (but only some of the appendices).

  2. I spent some time thinking about potential counterexamples. It's hard to say how much; this mostly wasn't time I carved out, more something I was thinking about while taking a walk or something.

  3. At times I would reread specific parts of the writeup that seemed important for thinking about whether a particular idea was

... (read more)
Congrats on submitting proposals [] that would have won you a $15,000 prize if you had been eligible! How long did it take you to come up with these proposals?
Comments for shorter Cold Takes pieces

[Placeholder for How artistic ideas could get harder to find comments]

1Kai Williams6mo
I generally like the innovation-as-mining hypothesis with regards to the science and with some respect to the arts, but I think that there is one issue with the logical chain. You said that "[i]f not for this phenomenon [that ideas get harder to find], sequels should generally be better than the original," but I don't think this is necessarily true. I think a more likely reason that sequels aren't generally better than the original is mostly regression to the mean and selection effects, with two main causes: 1. Pure quality: Presumably, an author or a screenwriter will only make a sequel if the original enjoyed a sufficient level of success to merit the effort. While some of that quality is likely due to the skill of the author, some of it is likely due to luck. Accordingly, the quality of subsequent sequels is likely worse, even if the author improves over time. 2. Idea selection: When writing the original, the author had a lot of leeway in what type of media was being written, what the world building looked like, what plot was going to be used, etc. Given the uncertainty of the enterprise and the need to make a good pitch, the author likely chose ideas for their highest quality. However, when the sequel is written, the choice of ideas is less strict. For one, the pool of ideas is smaller, but further, the standards are lower. If a sequel is written, it is often due to demand from audiences rather than the desire of creators, so ideas are not held to the same standard. I think a relevant example here is that of albums. There is this idea of a "sophomore slump" in albums, where a band's second album tends to be worse than their first. I don't think this is due to it being hard to make good albums after your first (quality generally seems to improve over the next few albums after that), but a shrinking pool of songs to choose from. On an artists debut album, they can choose pretty much any song they've ever written
I think this is one of your best posts. I learned a lot, built new models of art, and laughed out loud multiple times.
Obligatory link to Scott Alexander's "Ambijectivity" [] regarding the contentiousness of defining great art.
Forecasting transformative AI: the "biological anchors" method in a nutshell

My understanding is that "mixture of experts" essentially comes down to training multiple distinct models, and having some "meta" procedure for assigning problems (or pieces of problems) to them.

Since training expense grows with something like the square of model size, it's much more expensive to train one big model than N smaller models that are each 1/N as big (plus a procedure for choosing between the N smaller models).

A human brain is about 100x the "size" of a mouse brain. So for a metaphor, you can think of "mixture of experts" as though it's trying ... (read more)

1Douglas Knight4mo
It may be hard to compare art from different periods, but it is direct to compare science and engineering from different periods because the same thing was discovered or invented multiple times. Knowledge is not a ratchet. Sometimes knowledge is lost. But it is not only catastrophes like burning libraries and riots against scholars. There are Leaden Ages where scientific knowledge is lost century after century, such as Alexandria for about five centuries starting 150AD. Any period of progress is a Golden Age compared to that. Do people know that they are in a Leaden Age? I don't think the Alexandrians knew. The first task is not to fool yourself. If a second age reconstructs the knowledge of the first age faster, it might be because they are better, or it might be because they are supported by the notes of the pioneers. But what if they are slower? This is strong evidence that the first age really was Golden. In particular, the Hellenistic Age, 330-130BC, subsumed virtually all scientific progress for thousands of years, at least to 1600, and maybe to 1700.
One possibility (which may or not have been mentioned) is that an overflow of information/stimulation as a result of technology and faster paced societies inhibits creativity. Part of the issue may arise from excessive entertainment: Beethoven may have created musical pieces in periods of boredom, which the modern day Beethoven spends scrolling through social media or watching Netflix.

I was pretty struck by how per capita output isn't obviously going down, and it's only when you do the effective population estimates that it does.

Could this suggest a 4th hypothesis: the 'innate genius' theory: about 1 in 10 million people are geniuses, and at least since around 1400, talent spotting mechanisms were good enough to find them, so the fraction of the population that was educated or urbanised doesn't make a difference to their chances of doing great work. 

I think I've seen people suggest this idea - I'm curious why you didn't include it in the post.

1Stuart Carter6mo
I would argue that John Williams fits the bill of a modern Beethoven, but he's not much of an innovator. Jacob Collier innovates, but lacks mainstream appeal. Kanye West innovated hip hop quite a bit, but lacks appeal (in general) to a high-brow audience because he doesn't sing well or play any instruments, he's just really good at stitching together samples and surrounding himself with people that can refine his ideas. I think much of it has to do with the volume of artists and scientists and the increased flow of information - so that one single person is responsible for much less of the progress, because much more information is circulating about what that person is doing among that particular community. I think as far as academia is concerned you can also consider a sort of bureaucratic weight to things. It's harder to get anything done when you have to spend much of that time applying for grants, doing relatively asinine trainings for the purpose of giving your institution liability shielding and federal government funding, etc. Another thing is that there's so many relevant people that the towering figures that used to exist are rarer. Many of the most relevant figures before the modern day were aristocrats, whereas today it's much more of an open field in terms of who gets to do what - and so there are a lot of people competing for a relatively small amount of attention. If someone solved the riemann hypothesis tomorrow, I doubt the average person would hear much about it. Thoughts?
>the "golden age" hypothesis (people in the past were better at innovation), the "bad taste" hypothesis (Beethoven and others don't deserve their reputations), and the "innovation as mining" hypothesis (ideas naturally get harder to find [] over time, and we should expect art and science to keep slowing down by default). I think you're missing what I consider the most likely explanation: There are a lot more people in these fields now, trying to be the best. What's remarkable about these historical figures is not that they were better at what they did than people nowadays, but that they did it first. So I am not sure we'd notice a new Shakespeare. We'd simply lump him in with all of the other really good playwrights we have. Nothing would make him stand out as the best. So it's possible that our scientists, artists, etc. are better than these historical giants, but we just can't tell.
[Disclaimer: Sheer idle speculation, not important or rigorous] I am generally a fan of the innovation-as-mining hypothesis. However, even within the broad tent of that hypothesis, there is room to debate e.g. whether there has been a recent, temporary slowdown in progress due to cultural or genetic factors in addition to the usual ideas-getting-harder-to-find factor. I have two ideas here that I'd be interested to see explored: 1. You say What about a duplicate of John von Neumann? Maybe our modern geniuses like Terry Tao are his equal, but I sometimes wonder if he was a class above even them. 2. One argument you make against the Golden Age hypothesis is that typically the golden age is also the first age, which is a suspicious coincidence. IIRC, I read somewhere that average human brain size has shrunk over the last ten thousand years or so. I dunno if that's true but suppose it is. Given the correlation between brain size and IQ, one might wonder whether selection pressure for intelligence -- or some important component of it -- has also diminished in the last ten thousand years or so. If that were true, a version of the Golden Age hypothesis would be more likely, and also would successfully predict that observed "golden ages" in various fields would happen at the beginning of said fields.
The movie Yesterday sort of tackled this in an interesting way. Imagine a parallel universe where everything is the same but the Beatles never came together. Would someone releasing their exact music in 2021 still become highly successful and considered a musical icon? In the movie the answer is yes. In real life I imagine the answer would be no - the same exact music would no longer sound innovative and would thus not become particularly successful. This New-Beatles band might reach the level of a Top-100 artist but they'd never see the same level of admiration as the Beatles did and still do. So I believe we're simply not judging more recent art works by the same standards, resulting in a huge bias towards older works. Beethoven is only noteworthy because his works are a cultural meme at this point - he was a great musician for his time, sure, but right now there's probably tens of thousands of musicians who could make music of the same caliber straight on their laptops. Today's Beethoven publishes his amazing tracks on SoundCloud and toils in obscurity.
2John Loder6mo
Very much enjoyed the post. The thesis that recent (50 year) declines in innovation productivity are best explained by innovation generally getting structurally harder over time does, I think, somewhat overfit the data. Sketched argument below: 1. Innovation is cumulative. And in particular new tools create new possibilities for innovation as much as the reverse. So no astronomy without the telescope, no modern medicine without organic chemistry, no Beethoven without the invention of the piano, no early mathematics without Hindu-Arabic numerals, etc. 2. When the right tool arrives, there is a subsequent explosion of innovation, followed by a slow down. 3. There is a degree of randomness in these bursts, and the 70 years around the turn of the 19th/20th century was a particularly strong cluster (from the publication of Maxwell's equations in 1865 to the Trinity nuclear test in 1945). Humanity went from candles and horses to nuclear power, jet engines, eradication of most communicable diseases, electrification, relativity and quantum mechanics, the telephone, early computers, and many others. Art and culture also shifted abruptly and in a very interesting way. 4. Note that this was an acceleration from the 19th century - innovation doesn't always get harder. 5. If the limiting factor is the right tool, rather than people or money, then huge investment in research will lead to drops in productivity in producing fundamental breakthroughs. And the people we call geniuses are just those that get their hands on the tool first (bit like Bill Gates being one of a handful of people to globally able play with computers in their teens). 6. Post 1970 (?) slowdown in innovation is to some extent a contrast with an exceptional cluster, and and in itself a relative trough. The big question, it seems to me, is whether AI and ~CRISPR the sorts of fundamental tools that can spark a new acceleration?
"Some of the people who have written the most detailed pieces about "innovation stagnation" seem to believe something like the "golden age" hypothesis - but they seem to say so only in interviews and casual discussions, not their main works." Just fyi - You mention Peter Thiel in a footnote here. It's been a while since I read it but iirc Peter Thiel describes something you might consider a version of the golden age hypothesis in a bit of dusk in the "You are not a lottery ticket" chapter of zero to one.
I think another potential explanation relates to the way people think about the history of a given field when asked to reflect on it (e.g. to create a top 100 list). We tend to conceive of fields as progressions unfolding over time, and even if we don't think this is always in a "better" direction, at least we conceive of the field as consisting of time periods characterized by a dominant paradigm or style. Certainly this is the way that "history of X" classes are usually taught. If this is the case, it seems natural to me that, when asked to reflect on the "most important" individuals or contributions to a field in its history, we will tend to structure that reflection around our conception of these periods, and likely identify an emblematic individual for each period. Indeed, part of our conception of "greatest" might include a feature like "dominated their field for a decade or more," and obviously the frequency of individuals characterized in this way cannot increase with greater population, education, or anything at all! To the extent that our thinking follows this approach, we will tend to see "best of" lists being pretty flat over time, and therefore, appearing to decline when normalized by anything that increases over time.
1Jeff Sackmann6mo
Thanks for this, it's a fascinating subject. At risk of anticipating your follow-ups, I have two suggestions regarding art. I don't think they apply as well to science. 1. If a work is considered to be among the greats, the older it is, the more foundational it has become. An enormous amount of great music since Beethoven is, often very deliberately, developing Beethoven's ideas further, or introducing new ideas by tweaking what Beethoven (or Mozart or Bach) did. Thus, what the art is gets tied up with the foundational works. In mining terms, finding a motherlode also seems to mean shutting down (or, at best, reducing focus on) other mines. It's impossible to imagine western classical music without Beethoven, in part because such a significant amount of it is Beethovian. Had some very talented and charismatic musician come along at the right time from the Balkans, maybe that foundational slot would be taken by someone/something else. If this is correct, there's bound to be some historical figures that are considered head-and-shoulders above the rest, and they must be quite old. A contemporary person cannot fill this role, though it's conceivable that a contemporary person would fill this role for people 200 years down the road. 2. The effectiveness of Beethoven's and Shakespeare's works relies on performance. While there are attempts at period authenticity, the most popular recording of the 5th symphony or performance of Hamlet is not that, and hasn't been that for a long time. This relates to (1) in a couple of ways: 1. There have been centuries of "testing" to optimize the experience of these works, and it is ongoing. (This point is less relevant to, say, novelists, even if people are constantly re-interpreting Dickens.) Ranking Shakespeare #1 is really ranking centuries-of-optimizations-Shakespeare #1, which puts David Mamet at a pretty big disadvan
Thank you for a thought provoking post! I enjoyed it a lot. I also find the "innovation as mining" hypothesis intuitive. I'd just add that innovation gets harder for humans, but we don't know whether it holds in general (think AI). Our mental capacity has been roughly constant since ancient Greece, but there is more and more previous work to understand before one can come up with something new. This might not be true for AI, if their capacity scales. On the other hand there is a combinatorial explosion of facts that you can combine to come up with an innovation and I don't know what fraction of the combinations will actually be useful and judged as innovation. So overall, the difficulty might increase, stay roughly the same, or decrease, depending on how the number of useful combination scales with the number of all combinations. One explanation of this would be that innovation needs time to collect its impact. Old innovations are well integrated into the society, so they have already collected most of its impact, while new innovations have most of their impact still in the future, so we don't perceive them as transformative yet.
Summary of history (empowerment and well-being lens)

I broadly agree that my summary has this issue. If there were causal stories I were confident in, I would try to include them in the summary; but in fact I feel very hazy on a lot of multiple-step causal stories about history, and have defaulted to leaving them out when I think the case is quite unclear. I'm sure this leaves my summary less informative than it would ideally be (and than it would be if I knew more about history and were more confident about some of these multiple-step causal stories).

Bayesian Mindset

I agree with what you say here, as a general matter. I'm not sure I want to make an edit, as I really do think there are some "bad" parts of myself that I'd prefer to "expose and downweight," but I agree that it's easy to get carried away with that sort of thing.

Cool. We are on the same page then. :) I also agree that there are some bad parts of myself I'd prefer to expose and downweight.
Bayesian Mindset

It might be true that the right expected utility calculation would endorse being overconfident, but "Bayesian mindset" isn't about behaving like a theoretically ideal utility maximizer - it's about actually writing down probabilities and values and taking action based on those. I think trying to actually make decisions this way is a very awkward fit with an overconfident attitude: even if the equation you write down says you'll do best by feeling overconfident, that might be tough in practice.

The tension between overconfidence and rigorous thinking is overrated: Source and previous discussion [].
Load More