All of SammyDMartin's Comments + Replies

How should Effective Altruists think about Leftist Ethics?

I think that the mainstream objections from 'leftist ethics' are mostly  best thought of as claims about politics and economics that are broadly compatible with Utilitarianism but have very different views about things like the likely effects of charter cities on their environments - so if you want to take these criticisms seriously then go with 3, not 2.

There are some left-wing ideas that really do include different fundamental claims about ethics (Marxists think utilitarianism is mistaken and a consequence of alienation) - those could be addressed b... (read more)

Good news on climate change

I see - that seems really valuable and also exactly the sort of work I was suggesting (I.e. addressing impact uncertainty as well as temperature uncertainty).

In the meantime, are there any sources you could point me to in support of this position, or which respond to objections to current economic climate models?

Also, is your view that the current Econ models are fundamentally flawed but that the economic damage is still nowhere near catastrophic, or that those models are actually reasonable?

2mchr3k25dThere are a couple of sources which I'd recommend taking a look at. * 2015 Climate Change Risk Assessment [https://www.cser.ac.uk/media/uploads/files/climate-change--a-risk-assessment-v11.pdf] - particularly worth reading the section which includes the quote "A simple conclusion is that we need to know more about the impacts associated with higher degrees of temperature increase. But in many cases this is difficult. For example, it may be close to impossible to say anything about the changes that could take place in complex dynamic systems, such as ecosystems or atmospheric circulation patterns, as a result of very large changes very far into the future." * Climate change risk assessment 2021 [https://www.chathamhouse.org/sites/default/files/2021-09/2021-09-14-climate-change-risk-assessment-quiggin-et-al.pdf] - conclusion starts out by saying "Unless NDCs are dramatically increased, and policy and delivery mechanisms commensurately revised, many of the climate change impacts described in this paper are likely to be locked in by 2040, and become so severe they go beyond the limits of what nations can adapt to." Section 4 of this paper considers various cascading systemic risks. These are a big part of what makes predicting the future impacts of climate change so hard.

For sources, I would recommend just reading the technical summary of the 2014 IPCC Impacts report. There is no indication there that civilisation will end at 4 degrees.

I think a lot of the economic models are very flawed yes. I think it is more useful to look at the impacts literature and try and make your own mind up from there. But I also think it is instructive that the most pessimistic models suggest that 4 degrees of climate change would leave us with something like a 400% increase in GDP compared to a counterfactual 900% increase without climate cha... (read more)

Good news on climate change

Firstly, on the assumption that the direct or indirect global catastrophic risk (defined as killing >10% of the global population or doing equivalent damage) of climate change depends on warming of more than 6 degrees, the global catastrophic risk from climate change is at least an order of magnitude lower than previously thought. If you think 4 degrees of warming would be a global catastrophic risk, then that risk is also considerably lower than previously thought: where once it was the most likely outcome, the chance is now arguably lower than 5%.

I... (read more)

7tcelferact1moWhat historical precedent do you have in mind here? The reason my intuitions initially would go in the opposite direction is a case study like invasive species in Australia [https://en.wikipedia.org/wiki/Invasive_species_in_Australia]. tl;dr is when an ecosystem has evolved holding certain conditions constant (in this case geographical isolation), and that changes fairly rapidly, even a tiny change like a European rabbit can have negative consequences well beyond what was foreseen by the folks who made the change. I won't pretend to be an expert on how analogous climate is to this example, but if someone wanted to shift my intuitions, a good way to start would be to convince me that, for some given optimistic economic forecast, the likelihood it has missed significant knock-on negative consequences of an X degree average rise in temperature is <50%.

Speaking for me personally and not Johannes. I strongly disagree with the claim that 3,4, 5 or 6 degrees of warming would do anything even remotely close to ending human civilisation or causing civilisational collapse. However, I don't think this post is the best place to discuss the question of climate impacts. I am working on a large report on that question which will be out next year.

The Phil Torres essay in Aeon attacking Longtermism might be good

One substantive point that I do think is worth making is that Torres isn't coming from the perspective of common-sense morality Vs longtermism, but rather a different, opposing, non-mainstream morality that (like longtermism) is much more common among elites and academics.

Yet this Baconian, capitalist view is one of the most fundamental root causes of the unprecedented environmental crisis that now threatens to destroy large regions of the biosphere, Indigenous communities around the world, and perhaps even Western technological civilisation itself.

When he... (read more)

Robin Hanson on the Long Reflection

I don't think Hanson would disagree with this claim (that the future is more likely to be better by current values, given the long reflection, compared to e.g. Age of Em). I think it's a fundamental values difference.

Robin Hanson is an interesting and original thinker, but not only is he not an effective altruist, he explicitly doesn't want to make the future go well according to anything like present human values.

The Age of Em, which Hanson clearly doesn't think is an undesirable future, would contain very little of what we value. Hanson says this, but it... (read more)

4RobinHanson2moI have values, and The Age of Em overall contains a great deal that I value, and in fact probably more of what I value than does our world today.
Forecasting Compute - Transformative AI and Compute [2/4]

Great post! You might be interested in this related investigation by the MTAIR project I've been working on, whch also attempts to build on Ajeya's TAI timeline model, although in a slightly different way to yours (we focus on incorporating non-DL based paths to TAI as well as trying to improve on the 'biological anchors' method already described): https://forum.effectivealtruism.org/posts/z8YLoa6HennmRWBr3/link-post-paths-to-high-level-machine-intelligence 

2lennart2moThanks, Sammy. Indeed this is related and very interesting!
Summary of history (empowerment and well-being lens)

One thing that your account might miss is the impact of ideas on empowerment and well-being down the line. E.g. it's a very common argument that Christan ideas about the golden rule motivated anti-slavery sentiment, so if the Roman empire hadn't spread Christianity across Europe then we'd have ended up with very different values.

Similarly, even if the content of ancient Greek moral philosophy wasn't directly useful to improve wellbeing, they inspired the Western philosphical tradition that led to Enlignment ideals that led to the abolition of slavery.

I've ... (read more)

3Holden Karnofsky2moI'd say some of both. * If I tried to start noting not just manifestly important changes in empowerment and well-being, but also earlier developments that might have been causally important for them, I think the project would get a lot more unwieldy and more packed with judgment calls, and I chose to mostly just refrain from doing that. * I am in fact skeptical by default of claims along the lines of: "Idea X was important for development Y, despite the observation idea X was around for centuries with little-to-no movement on Y, and then Y changed rapidly a very long time later."
Why AI alignment could be hard with modern deep learning

Very good summary! I've been working on a (much drier) series of posts explaining different AI risk scenarios - https://forum.effectivealtruism.org/posts/KxDgeyyhppRD5qdfZ/link-post-how-plausible-are-ai-takeover-scenarios

But I think I might adopt 'Sycophant'/'Schemer' as better more descriptive names for WFLL1/WFLL2, Outer/Inner alignment failure going forward

I also liked that you emphasised how much the optimist Vs pessimist case depends on hard to articulate intuitions about things like how easily findable deceptive models are and how easy incremental co... (read more)

AMA: Jason Brennan, author of "Against Democracy" and creator of a Georgetown course on EA

Thanks for this reply. Would you say then that Covid has strengthened the case for some sorts of democracy reduction, but not others? So we should be more confident in enlightened preference voting but less confident in Garett Jones' argument (from 10% less democracy) in favour of more independent agencies?

AMA: Jason Brennan, author of "Against Democracy" and creator of a Georgetown course on EA

Do you think that the West's disastrous experience with Coronavirus (things like underinvesting in vaccines, not adopting challenge trials, not suppressing the virus, mixed messaging on masks early on, the FDA's errors on testing, and others as enumerated in this thread- or in books like The Premonition) has strengthened, weakened or not changed much the credibility of your thesis in 'Against Democracy', that we should expect better outcomes if we give the knowledgeable more freedom to choose policy?

For reasons it might weaken 'Against Democracy', it seems... (read more)

8Jason Brennan3moI think it has in some ways strengthened my overall philosophy. I've been pushing public choice ideas for a while, and the FDA and CDC seemed to band together this year to make that look right. Epistocracy should not be confused with technocracy. In a technocracy, a small band of experts get lots of power to manipulate people, nudge them, or engage in social engineering. Many democrats are technocrats--indeed, the people I argue with, like Christiano, Estlund, and so on, are pretty hardcore technocrats who have been in favor of letting alphabet agencies have lots of dictatorial power during this crisis. Instead, epistocracy is about weighing votes during elections to try to produce better electoral or referendum results. For instance, I favor a system of enlightened preference voting where we let everyone vote but we then calculate what the public would have supported had it been fully informed. And there is decent evidence that if we used it, one thing that would happen is that the resulting voting public would be more aware of the limitations of technocrats and would be more in favor of civil liberties.
3Stefan_Schubert3moYou may want to specify in what sense Western countries' experience with Covid has been disastrous, in your view (and how you think policy should have been different).
Does Moral Philosophy Drive Moral Progress?

I don't think the view that moral philosophers had a positive influence on moral developments in history is a simple model of 'everyone makes a mistake, moral philosopher points out the mistake and convinces people, everyone changes their minds'. I think that what Bykvist, Ord and MacAskill were getting at is that these people gave history a shove at the right moment.

At the very least, it doesn't seem that discovering the correct moral view is sufficient for achieving moral progress in actuality.

I have no doubt that they'd agree with you about this. But if... (read more)

2AppliedDivinityStudies5moThanks, this is a good comment.
COVID: How did we do? How can we know?

Is there any public organisation which can be proud of last year?

This is an important question, because we want to find out what was done right organizationally in a situation where most failed, so we can do more of it. Especially if this is a test-run for X-risks.

There are two examples that come to mind of government agencies that did a moderately good job at a task which was new and difficult. One is the UK's vaccine taskforce, which was set up by Dominic Cummings and the UK's chief scientific advisor, Patrick Vallance and responsible for the relatively ... (read more)

3ChristianKleineidam5moFast rollout would be vaccinating every wiling citizen by July or August 2020. That what you would have gotten if you would have asked who can provide the vaccines by that point, put up prices and get the regulatory barriers out of the way. Stoecker vaccinated himself in March 2020. His company was then selling COVID-19 tests because it was allowed to do so. If it would have been allowed to sell vaccines with regulations that are as easy as selling tests it would likely also have sold those. That task force might have gotten a D and not an F but calling it a moderately good job is a huge stretch.

We even saw an NYT article about the CDC and whether reform is possible.

There were some other recent NYT articles which based on my limited COVID knowledge I thought were pretty good, e.g. on the origin of the virus or airborne vs. droplet transmission [1].

The background of their author, however, seems fairly consistent with an "established experts and institutions largely failed" story:

Zeynep Tufekci, a contributing opinion writer for The New York Times, writes about the social impacts of technology. She is an assistant professor in the School of Informat

... (read more)
Why should we *not* put effort into AI safety research?

Alignment by default: if we have very strong reasons to expect that the methods that are best suited for ensuring that AI is aligned are the same as the methods that are best suited for ensuring that we have AI that is capable enough to understand what we want and act on it, in the first place.

To the extent that alignment by default is likely we don't need a special effort to be put into AI safety because we can assume that the economic incentives will be such that we will put as much effort into AI safety as is needed, and if we don't put the sufficient e... (read more)

Vitalik Buterin just donated $54M (in ETH) to GiveWell

We know he's been active on lesswrong in the past. Is it possible he's been reading the posts here?

Multiple people connected to the lesswrong/ea investing groups tried to contact him. We both contacted him directly and got some people closer to Vitalik to talk to him. I am unsure how much influence we had. He donated less than two days after the facebook threads went up.

We definitely tried!

Three charitable recommendations for COVID-19 in India

Thanks for getting back to me - I took Jeff's calculations and did some guesstimating to try and figure out what demand might look like over the next few weeks. The only covid forecast I was able to find for India (let me know if you've seen another!) is this by IHME. Their 'hospital resource use' forecast shows that they expect a demand of 2 million beds, roughly what was the case in the week before Jeff produced his estimate of the value of oxygen-based interventions (last week of April), to be exceeded until the start of June, which is 30 days from when... (read more)

3Tejas Subramaniam7moThanks a lot for this estimate! I will link your comment on our post.
Three charitable recommendations for COVID-19 in India

Thanks for getting this done so quickly! Do you have any internal estimates (even order of magnitude ones) of the margin by which this exceeds Givewell's top recommended charities? I'm intending to donate, but my decision would be significantly different if, for example, you thought GiveIndia Oxygen fundraiser was currently ~1-1.5 times better than Givewell's top recommended charities, versus ~20 times better.

4Tejas Subramaniam7moThanks for the comment! We are honestly quite unsure about the margin, especially because the cost-effectiveness analyses we have access to are about the cause area and not a model for the specific charitable donations. Our guess is that donations to oxygen likely beat GiveWell top charities – here are Jeff Coleman’s calculations for the GiveIndia’s various programs for magnitudes [https://twitter.com/technocrypto/status/1388947180753457159]. It’s hard to give a precise estimate partly because each oxygen concentrator or cylinder, for instance, is a fixed cost which can be used for a while (and we‘re unclear what demand will be like over the next few months) – so the best Coleman can do is an estimate of days of use it takes to beat GiveWell top charities. Our current guess is that the direct cash transfer program in our second recommendation is competitive with GiveDirectly but most other GiveWell-recommended charities beat it on pure cost-effectiveness terms, and we are much more uncertain about our third recommendation.
Moral Anti-Realism Sequence #3: Against Irreducible Normativity
I kind of feel this way, except that I think the target criteria can differ between people, and are often underdetermined. (As you point out in some comment, things also depend on which parts of one's psychology one identifies with.)

I think that you were referring to this?

Normative realism implies identification with system 2
...
I find this very interesting because locating personal identity in system 1 feels conceptually impossible or deeply confusing. No matter how much rationalization goes on, it never seems intuitive to identify myself with
... (read more)
3seanrson1yAFAIK the paralysis argument is about the implications of non-consequentialism, not about down-side focused axiologies. In particular, it's about the implications of a pair of views. As Will says in the transcript you linked: "but this is a paradigm nonconsequentialist view endorses an acts/omissions distinction such that it’s worse to cause harm than it is to allow harm to occur, and an asymmetry between benefits and harms where it’s more wrong to cause a certain amount of harm than it is right or good to cause a certain amount of benefit... And if you have those two claims, then you’ve got to conclude [along the lines of the paralysis argument]". Also, I'm not sure how Lukas would reply but I think one way of defending his claim which you criticize, namely that "the need to fit all one’s moral intuitions into an overarching theory based solely on intuitively appealing axioms simply cannot be fulfilled", is by appealing to the existence of impossibility theorems in ethics. In that case we truly won't be able to avoid counterintuitive results (see e.g. Arrhenius 2000, Greaves 2017). This also shouldn't surprise us too much if we agree with the evolved nature of some of our moral intuitions.
Center for Global Development: The UK as an Effective Altruist

I thought that this post would make a bigger deal of the UK's coronavirus response - currently top in the world for both vaccine development and large-scale clinical trials, and one of the leading funders of international vaccine development research.

Moral Anti-Realism Sequence #3: Against Irreducible Normativity

How to make anti-realism existentially satisfying

Instead of “utilitarianism as the One True Theory,” we consider it as “utilitarianism as a personal, morally-inspired life goal...
”While this concession is undoubtedly frustrating, proclaiming others to be objectively wrong rarely accomplished anything anyway. It’s not as though moral disagreements—or disagreements in people’s life choices—would go away if we adopted moral realism.

If your goal here is to convince those inclined towards moral realis... (read more)

4Lukas_Gloor1yNice point! Yeah, I think that's a good suggestion. I had a point about "arguments can't be unseen" – which seems somewhat related to the alienation point. I didn't quite want to imply that morality is just a life goal. There's a sense in which morality is "out there" – it's just more underdetermined than the realists think, and maybe more goes into whether or not to feel compelled to dedicate all of one's life to other-regarding concerns. I emphasize this notion of "life goals" because it will play a central role later on in this sequence. I think it's central to all of normativity. Back when I was a moral realist, I used to say "ethics is about goals" and "everything is ethics." There's this position "normative monism" that says all of normativity is the same thing. I kind of feel this way, except that I think the target criteria can differ between people, and are often underdetermined. (As you point out in some comment, things also depend on which parts of one's psychology one identifies with.)
Moral Anti-Realism Sequence #3: Against Irreducible Normativity

You've given me a lot to think about! I broadly agree with a lot of what you've said here.

I think that it is a more damaging mistake to think moral antirealism is true when realism is true than vice versa, but I agree with you that the difference is nowhere near infinite, and doesn't give you a strong wager.

However, I do think that normative anti-realism is self-defeating, assuming you start out with normative concepts (though not an assumption that those concepts apply to anything). I consider this argument to be step 1 in establishing mora... (read more)

3Lukas_Gloor1yThis discussion continues to feel like the most productive discussion I've had with a moral realist! :) [...] [...] I think I agree with all of this, but I'm not sure, because we seem to draw different conclusions. In any case, I'm now convinced I should have written the AI's dialogue a bit differently. You're right that the AI shouldn't just state that it has no concept of irreducible normative facts. It should provide an argument as well! What would you reply if the AI uses the same structure of arguments against other types of normative realism as it uses against moral realism? This would amount to the following trilemma for proponents of irreducible normativity (using section headings from my text): (1) Is irreducible normativity about super-reasons? (2) Is (our knowledge of) irreducible normativity confined to self-evident principles? (3) Is there a speaker-independent normative reality? I think you're inclined to agree with me that (1) and (2) are unworkable or not worthy of the term "normative realism." Also, it seems like there's a weak sense in which you agree with the points I made in (3), as it relates to the domain of morality. But maybe you only agree with my points in (3) in a weak sense, whereas I consider the arguments in that section to have stronger implications. The way I thought about this, I think the points in (3) apply to all domains of normativity, and they show that unless we come up with some other way to make normative concepts work that I haven't yet thought of, we are forced to accept that normative concepts, in order to be action-guiding and meaningful, have to be linked to claims about convergence in human expert reasoners. Doesn't this pin down the concept of irreducible normativity in a way that blocks any infinite wagers? It doesn't feel like proper non-naturalism anymore once you postulate this link as a conceptual necessity. "Normativity" became a much more mundane concept after we accepted this link. The trilemma appli
‘The Precipice’ Book Review
But instilling the urgency to do so may require another type of writing-that of science fiction, of more creative visionaries who are willing to paint in vivid detail a picture of what a flourishing human future could be.

If it's emotive force you're after, you may be interested in this - Toby Ord just released a collection of quotations on Existential risk and the future of humanity, everyone from Kepler to Winston Churchill (in fact, a surprisingly large number are from Churchill) to Seneca to Mill to the Aztecs - it's one of the most i... (read more)

AMA or discuss my 80K podcast episode: Ben Garfinkel, FHI researcher

Parfit isn't quite a non-naturalist (or rather, he's a very unconventional kind of non-naturalist, not a Platonist) - he's a 'quietist'. Essentially, it's the view that there are normative facts, they aren't natural facts, but we don't feel the need to say what category they fall into metaphysically, or that such a question is meaningless.

I think a variant of that, where we say 'we don't currently have a clear idea what they are, just some hints that they exist because of normative convergence, and the inte... (read more)

Moral Anti-Realism Sequence #3: Against Irreducible Normativity

This is an interesting post, and I have a couple of things to say in response. I'm copying over the part of my shortform that deals with this:

Normative Realism by degrees

Further to the whole question of Normative / moral realism, there is this post on Moral Anti-Realism. While I don't really agree with it, I do recommend reading it - one thing that it convinced me of is that there is a close connection between your particular normative ethical theory and moral realism. If you claim to be a moral realist but don't make ethical claims beyo... (read more)

3Lukas_Gloor1yCool, I'm happy that this argument appeals to a moral realist! I agree that it then shifts the arena to convergence arguments. I will discuss them in posts 6 and 7. In short, I don't think of myself as a moral realist because I see strong reasons against convergence about moral axiology and population ethics. I don't think this argument ("anti-realism is self-defeating") works well in this context. If anti-realism is just the claim "the rocks or free-floating mountain slopes that we're seeing don't connect to form a full mountain," I don't see what's self-defeating about that. One can try to say that a mistaken anti-realist makes a more costly mistake than a mistaken realist. However, on close inspection, I argue that this intuition turns out to be wrong. It also depends a lot on the details. Consider the following cases: (1) A person with weak object-level normative opinions. To such a person, the moral landscape they're seeing looks like either: (1a) free-floating rocks or parts of mountain slope, with a lot of fog and clouds. (1b) many (more or less) full mountains, all of which are similarly appealing. The view feels disorienting. (2) A person with strong object-level normative opinions. To such a person, the moral landscape they're seeing looks like either: (2a) a full mountain with nothing else of note even remotely in the vicinity. (2b) many (more or less) full mountains, but one of which is definitely theirs. All the other mountains have something wrong/unwanted about them. 2a is confident moral realism. 2b is confident moral anti-realism. 1a is genuine uncertainty, which is compatible with moral realism in theory, but there's no particular reason to assume that the floating rocks would connect. 1b is having underdefined values. Of course, how things appear to someone may not reflect how they really are. We can construct various types of mistakes that people in the above examples might be making. This requires longer discussion, but I feel
AMA or discuss my 80K podcast episode: Ben Garfinkel, FHI researcher

Hi Ben,

Thanks for the reply! I think the intuitive core that I was arguing for is more-or-less just a more detailed version of what you say here:

"If we create AI systems that are, broadly, more powerful than we are, and their goals diverge from ours, this would be bad -- because we couldn't stop them from doing things we don't want. And it might be hard to ensure, as we're developing increasingly sophisticated AI systems, that there aren't actually subtle but extremely important divergences in some of these systems' goal
... (read more)
2Ben Garfinkel1yQuick belated follow-up: I just wanted to clarify that I also don't think that the orthogonality thesis or instrumental convergence thesis are incorrect, as they're traditionally formulated. I just think they're not nearly sufficient to establish a high level of risk, even though, historically, many presentations of AI risk seemed to treat them as nearly sufficient. Insofar as there's a mistake here, the mistake concerns way conclusions have been drawn from these theses; I don't think the mistake is in the theses themselves. (I may not stress this enough in the interview/slides.) On the other hand, progress/growth eventually becoming much faster might be wrong (this is an open question [https://www.nber.org/papers/w23928] in economics). The 'classic arguments' also don't just predict that growth/progress will become much faster. In the FOOM debate, for example, both Yudkowsky and Hanson start from the position that growth will become much faster; their disagreement is about how sudden, extreme, and localized the increase will be. If growth is actually unlikely to increase in a sudden, extreme, and localized fashion, then this would be a case of the classic arguments containing a "mistaken" (not just insufficient) premise.
AMA or discuss my 80K podcast episode: Ben Garfinkel, FHI researcher

You said in the podcast that the drop was 'an order of magnitude', so presumably your original estimate was 1-10%? I note that this is similar to Toby Ord's in The Precipice (~10%) so perhaps that should be a good rule of thumb: if you are convinced by the classic arguments your estimate of existential catastrophe from AI should be around 10% and if you are unconvinced by specific arguments, but still think AI is likely to become very powerful in the next century, then it should be around 1%?

4Ben Garfinkel1yThose numbers sound pretty reasonable to me, but, since they're roughly my own credences, it's probably unsurprising that I'm describing them as "pretty reasonable" :) On the other hand, depending on what counts as being "convinced" of the classic arguments, I think it's plausible they actually support a substantially higher probability. I certainly know that some people assign a significantly higher than 10% chance to an AI-based existential catastrophe this century. And I believe that Toby's estimate, for example, involved weighing up different possible views.
AMA or discuss my 80K podcast episode: Ben Garfinkel, FHI researcher

Hi Ben - this episode really gave me a lot to think about! Of the 'three classic arguments' for AI X-risk you identify, I argued in a previous post that the 'discontinuity premise' is based on taking a high-level argument that should be used to establish that sufficiently capable AI will produce very fast progress too literally and assuming the 'fast progress' has to happen suddenly and in a specific AI.

Your discussion of the other two arguments led me to conclude that the same sort of mistake is at work in all of them, as I e... (read more)

2Ben Garfinkel1yHi Sammy, Thanks for the links -- both very interesting! (I actually hadn't read your post before.) I've tended to think of the intuitive core as something like: "If we create AI systems that are, broadly, more powerful than we are, and their goals diverge from ours, this would be bad -- because we couldn't stop them from doing things we don't want. And it might be hard to ensure, as we're developing increasingly sophisticated AI systems, that there aren't actually subtle but extremely important divergences in some of these systems' goals." At least in my mind, both the classic arguments and the arguments in "What Failure Looks Like" share this common core. Mostly, the challenge is to explain why it would be hard to ensure that there wouldn't be subtle-but-extremely-important divergences; there are different possible ways of doing this. For example: Although an expectation of discontinous (or at least very fast) progress is a key part of the classic arguments, I don't consider it part of the intuitive core; the "What Failure Looks Like" picture doesn't necessarily rely on it. I'm not sure if there's actually a good way to take the core intuition and turn it into a more rigorous/detailed/compelling argument that really works. But I do feel that there's something to the intuition; I'll probably still feel like there's something to the intuition, even if I end feeling like the newer arguments have major issues too. [[Edit: An alternative intuitive core, which I sort of gesture at in the interview, would simply be: "AI safety and alignment issues exist today. In the future, we'll have crazy powerful AI systems with crazy important responsibilities. At least the potential badness of safety and alignment failures should scale up with these systems' power and responsibility. Maybe it'll actually be very hard to ensure that we avoid the worst-case failures."]]