I didn’t down vote but it seems like you are attacking a straw man here… the book is explicitly focused on the conditional IF anyone builds it. They never claim to know how to build it but simply suggest that it is not unlikely to be built in the future. I don’t know in which world you are living but this starting assumption seems pretty plausible to me (and quite a few other people more knowledgeable than me on those topics such as Nobel prize and Turing Award winners…). If not in 5 then maybe in 50 years. 

I would say at this point the burden is on you to make the case that the overall topic is nothing to worry about. Why not write your own book or posts where you let your arguments speak for themselves? 

So, you do it on purpose, not out of inability? Thanks for clarifying.

I love this question and I am looking forward to see what hedonic utilitarians come up with here. This has similar vibes to computronium thought experiments but better. Thanks for pointing this question out to me :)

Thanks for sharing this! It's an entertaining read and a valuable reminder of the limits of our perspectives. I love how the cleaner shows up at the end. True koan vibes!

I don't have time to read the full post and series but the logic of your argument reminds me very much of Werner Ulrich's work. May be interesting for you to check him out. I will list suggested references in order of estimated cost/benefit. The first paper is pretty short but already makes some of your key arguments and offers a proposal for how to deal with what you call "unawareness". 

Ulrich, W. (1994). Can We Secure Future-Responsive Management Through Systems Thinking and Design? Interfaces, 24(4), 26–37. https://doi.org/10.1287/inte.24.4.26

Ulrich, W. (2006). Critical Pragmatism: A New Approach to Professional and Business Ethics. In Interdisciplinary Yearbook for Business Ethics. V. 1, v. 1,. Peter Lang Pub Inc.

Ulrich, W. (1983). Critical heuristics of social planning: A new approach to practical philosophy. P. Haupt.

I think it would be helpful to not use longtermism in this synonymous way because I think it’s prone to lead to misunderstandings and unproductive conflict. 

For example, there is a school of thought called the person affecting view, which denies that future, non-existing people have moral patient hood but would still be able to have reasonable discussions about intergenerational justice in the sense of children might want to have children, etc. 

In general, I wouldn’t characterize those views as any more or less extreme or flat-footed than weak forms of longtermism. I think these are difficult topics that are contentious by nature. 

For me, the key is to stay open-minded and seek some form of discursive resolution that allows us to move forward in a constructive and ideally for all acceptable way. (That’s a critical pragmatist stance inspired by discourse ethics)

This is why I appreciate your curiosity and willingness to engage with different perspectives, even if it’s sometimes hard to understand opposing viewpoints. Keep at it! :)

It seems to me that you use “intergenerational justice” and longtermism in somewhat synonymous fashion. I think I would disagree with this sentiment. Longtermism is a specific set of positions whereas I would see intergenerational justice as a more open concept that can be defined and discussed from different positions.

I also think that there are reasonable critiques of longtermism. In the spirit of your post, I hope you stay open to considering those views.

I have only read the summarybot comment but based on that I wanted to leave a literature suggestion that could be interesting to people who liked this post and want to think more about how to put a pragmatic approach to ethics into practice.

Ulrich, W. (2006). Critical Pragmatism: A New Approach to Professional and Business Ethics. In Interdisciplinary Yearbook for Business Ethics. V. 1, v. 1,. Peter Lang Pub Inc. 

Abstract: Major contemporary conceptions of ethics such as discourse ethics and neocontractarian ethics are not grounded in a sufficiently pragmatic notion of practice. They suffer from serious problems of application and thus can hardly be said to respond to the needs of professionals and decision makers. A main reason lies in the tendency of these approaches to focus more on the requirements of ethical universalization than on those of doing justice to particular contexts of action – at the expense of practicality and relevance to practitioners. If this diagnosis is not entirely mistaken, a major methodological challenge for professional and business ethics consists in finding a new balance between ethical universalism and ethical contextualism. A reformulation of the pragmatic maxim (the methodological core principle of American pragmatism) in terms of systematic boundary critique (the methodological core principle of the author’s work on critical systems thinking and reflective professional practice) may provide a framework to this end, critical pragmatism.

I am wondering if you could say something about how the political developments in the US (i.e., Trump 2.0) are affecting your thinking on AGI race dynamics? It seems like the default assumption communicated publicly is still that the US are "the good guys" and a "western liberal democracy" that can be counted on, when the actual actions on the world stage are casting at least some doubt on this position. In some sense, one could even argue that we are already playing out a high-stakes alignment crisis at this very moment.

Any reactions or comments on this issue? I understand that openness around this topic is difficult at the moment but I also don't think that complete silence is all that wise either.