Matthew_Barnett's Quick takes

I want to say thank you for holding the pole of these perspectives and keeping them in the dialogue. I think that they are important and it's underappreciated in EA circles how plausible they are.

(I definitely don't agree with everything you have here, but typically my view is somewhere between what you've expressed and what is commonly expressed in x-risk focused spaces. Often also I'm drawn to say "yeah, but ..." -- e.g. I agree that a treacherous turn is not so likely at global scale, but I don't think it's completely out of the question, and given that I think it's worth serious attention safeguarding against.)

In particular, I am persuaded by the argument that, because evaluation is usually easier than generation, it should be feasible to accurately evaluate whether a slightly-smarter-than-human AI is taking unethical actions, allowing us to shape its rewards during training accordingly. After we've aligned a model that's merely slightly smarter than humans, we can use it to help us align even smarter AIs, and so on, plausibly implying that alignment will scale to indefinitely higher levels of intelligence, without necessarily breaking down at any physically realistic point.

This reasoning seems to imply that you could use GPT-2 to oversee GPT-4 by bootstrapping from a chain of models of scales between GPT-2 and GPT-4. However, this isn't true, the weak-to-strong generalization paper finds that this doesn't work and indeed bootstrapping like this doesn't help at all for ChatGPT reward modeling (it helps on chess puzzles and for nothing else they investigate I believe).

I think this sort of bootstrapping argument might work if we could ensure that each model in the chain was sufficiently aligned and capable of reasoning such that it would carefully reason about what humans would want if they were more knowledgeable and then rate outputs based on this. However, I don't think GPT-4 is either aligned enough or capable enough that we see this behavior. And I still think it's unlikely it works under these generous assumptions (though I won't argue for this here).

In fact, it is difficult for me to name even a single technology that I think is currently underregulated by society.

The obvious example would be synthetic biology, gain-of-function research, and similar.

I also think AI itself is currently massively underregulated even entirely ignoring alignment difficulties. I think the probability of the creation of AI capable of accelerating AI R&D by 10x this year is around 3%. It would be extremely bad for US national interests if such an AI was stolen by foreign actors. This suffices for regulation ensuring very high levels of security IMO. And this is setting aside ongoing IP theft and similar issues.

Some takes:

• I think Holly's tweet was pretty unreasonable and judge her for that not you. But I also disagree with a lot of other things she says and do not at all consider her to speak for the movement
• To the best of my ability to tell (both from your comments and private conversations with others), you and the other Mechanize founders are not getting undue benefit from Epoch funders apart from less tangible things like skills, reputation, etc. I totally agree with your comment below that this does not seem a betrayal of their trust. To me, it seems more a mutually beneficial trade between parties with different but somewhat overlapping values, and I am pro EA as a community being able to make such trades.
• AI is a very complex uncertain and important space. This means reasonable people will disagree on the best actions AND that certain actions will look great under some worldviews and pretty harmful under others
• As such, assuming you are sincere about the beliefs you've expressed re why to found Mechanize, I have no issue with calling yourself an Effective Altruist - it's about evidence based ways to do the most good, not about doing good my way

Separately:

• Under my model of the world, Mechanize seems pretty harmful in a variety of ways, in expectation
• I think it's reasonable for people who object to your work to push back against it and publicly criticise it (though agree that much of the actual criticism has been pretty unreasonable)
• The EA community implicitly gives help and resources to other people in it. If most people in the community think that what you're doing is net harmful even if you're doing it with good intentions, I think it's pretty reasonable to not want to give you any of that implicit support?

Matthew's comment was on -1 just now. I'd like to encourage people not to vote his post into the negative. Even though I don't find his defense at all persuasive, I still think it deserves to be heard. 

What I perceive as a measured, academic disagreement

This isn't merely an "academic disagreement" anymore. You aren't just writing posts, you've actually created a startup. You're doing things in the space.

As an example, it's neither incoherent nor hypocritical to let philosophers argue "Maybe existence is negative, all things considered" whilst still cracking down on serial killers. The former is necessary for academic freedom, the latter is not.

The point of academic freedom is to ensure that the actions we take in the world are as well-informed as possible. It is not to create a world without any norms at all.

It appears that advocating for slowing AI development has become a "sacred" value... Such reactions frankly resemble the behavior of a cult

Honestly, this is such a lazy critique. Whenever anyone disagrees with a group, they can always dismiss them as a "cult" or "cult-adjacent", but this doesn't make it true.

I think Ozzie's framing of cooperativeness is much more accurate. The unilateralist's curse very much applies to differential technology development, so if the community wants to have an impact here, it can't ignore the issue of "cowboys" messing things up by rowing in the opposite direction, especially when their reasoning seems poor. Any viable community, especially one attempting to drive change, needs to have a solution to this problem.

Having norms isn't equivalent to being a cult. When Fair Trade started taking off, I shared some of my doubts with some people who were very committed to it. This went poorly. They weren't open-minded at all, but I wouldn't run around calling Fair Trade a cult or even cult adjacent. They were just... a regular group.

And if I had run around accusing them of essentially being a "cult" that would have reflected poorly on me rather than on them. 

I have been publicly labeled a "sellout and traitor"... simply because I cofounded an AI startup

As I described in my previous comment, the issue is more subtle than this. It's about the specific context:

This is also a massive burning of the commons. It is valuable for forecasting/evals orgs to be able to hire people with a diversity of viewpoints in order to counter bias. It is valuable for folks to be able to share information freely with folks at such forecasting orgs without having to worry about them going off and doing something like this.

However, this only works if those less worried about AI risks who join such a collaboration don't use the knowledge they gain to cash in on the AI boom in an acceleratory way. Doing so undermines the very point of such a project, namely, to try to make AI go well. Doing so is incredibly damaging to trust within the community.

I concede that there wasn't a previous well-defined norm against this, but norms have to get started somehow. And this is how it happens, someone does something, people are like wtf and then, sometimes, a consensus forms that a norm is required.

Thanks for writing on the forum here - I think its brave of you to comment where there will obviously be lots of pushback. I've got a question relating to the new company and EA assignment. You may well have answered this somewhere else, if that's the case please point me in that direction. I'm a Global Health guy mostly, so am not super deep in AI understanding, so this question may be Naive.

Question: If we frame EA along the (great new website) lines of "Find the best ways to help others", how are you, through your new startup doing this? Is the for the purpose of earning to Give money away? Or do you think the direct work the startup will do has a high EV for doing lots of good? Feel free to define EA along different lines if you like!

 

I have been publicly labeled a "sellout and traitor" on X by a prominent member of the EA community simply because I cofounded an AI startup.

This accusation was not because you cofounded an AI startup. It was specifically because you took funding to work on AI safety from people who want to slow down AI development use capability trends to better understand how to make AI safer*, and you are now (allegedly) using results developed from that funding to start a company dedicated to accelerating AI capabilities.

I don't know exactly what results Mechanize is using, but if this is true, then it does indeed constitute a betrayal. Not because you're accelerating capabilities, but because you took AI safety funding and used the results to do the opposite of what funders wanted.

*Corrected to give a more accurate characterization, see Chris Leong's comment

Quick thoughts:
1. I think I want to see more dialogue here. I don't personally like the thought of the Mechanize team and EA splitting apart (at least, more than is already the case). I'd naively expect that there might still be a fair bit of wiggle room for the Mechanize team to do better or worse things in the world, and I'd of course hope for the better size of that. (I think the situation is still very early for instance). 
2. I find it really difficult to adjudicate on morality and specifics of the Mechanize spinnoff. I don't know as much about the details as others do. It really isn't clear to me what the previous funders of Epoch believed or what the conditions of the donations were. I think those details matter in trying to judge the situation.
3. The person you mentioned, Holly Elmore, is really the first and and one of the loudest to get upset about many things of this sort of shape. I think Holly disagrees with much of the EA scene, but in the opposite way than you/Matthew does. I personally think Holly goes a fair bit too far much of the time. That said, I know there were others who were upset about this who I think better represent the main EA crowd. 
4. "the idea that we should decelerate AI development is now sometimes treated as central to the EA identity in many (albeit not all) EA circles." The way I see it is more that it's somewhat a matter of cooperativeness between EA organizations. There are a bunch of smart people and organizations working hard to slow down generic AI development. Out of all the things one could do, there are many useful things to work on other than [directly speeding up AI development]. This is akin to how it would be pretty awkward if there were a group that calls themselves EA that tries to fight global population growth by making advertisements attacking GiveWell - it might be the case that they feel like they have good reasons for this, but it makes sense to me why some EAs might not be very thrilled. Related, I've seen some arguments for longer timelines that makes sense to me, but I don't feel like I've seen many arguments in favor of speeding up AI timelines that make sense to me. 

What do you think would constitute being a "sellout and traitor"?

Edit: I think that Neel's comment is basically just a better version of the stuff I was trying to say. (On the object level I'm a little more sympathetic than him to ways in which Mechanize might be good, although I don't really buy the story to that end that I've seen you present.)

Wanting to note that on my impressions, and setting aside who is correct on the object-level question of whether Mechanize's work is good for the world:

• My best read of the situation is that Matthew has acted very reasonably (according to his beliefs), and that Holly has let herself down a bit
  • I believe that Holly honestly feels that Matthew is a sellout and a traitor; however, I don't think that this is substantiated by reasonable readings of the facts, and I think this is the kind of accusation which it is socially corrosive to make publicly based on feelings
• On handling object-level disagreements about what's crucial to do in the world ...
  • I think that EA-writ-large should be endorsing methodology more than conclusions
  • Inevitably we will have cases where people have strong earnest beliefs about what's good to do that point in conflicting directions
    • I think that we need to support people in assessing the state of evidence and then acting on their own beliefs (hegemony of majority opinion seems kinda terrible)
    • Of course people should be encouraged to beware unilateralism, but I don't think that can extend to "never do things other people think are actively destructive"
  • It's important to me that EA has space for earnest disagreements
    • I therefore think that we should have something like "civilized society" norms, which constrain actions
      • Especially (but not only!) those which would be harmful to the ability for the group to have high-quality discourse
      • cf. SBF's actions, which I think were indefensible even if he earnestly believed them to be the best thing
      • (Some discussion on how norms help to contain naive utilitarianism)
    • I feel that Holly's tweet was (somewhat) norm-violating; and kind of angry that Matthew is the main person defending himself here

Just as our ancestors experienced before us, we face the prospect of losing the world we know in exchange for material progress and prosperity. 

Your ancestors who adopted agriculture did so because they thought that they and their children would get to eat the bread, not that they were sowing the seeds of their own destruction. If they had known that planting crops would lead to invasion and replacement they likely would not have done it. This rather large dis-analogy makes me think your use of the word 'just' is a bit of a stretch here.

If you and me and all of humanity gets killed by AI and turned into paperclips, that would be an unprecedented moral catastrophe. If the AIs that killed all of us stay around and enjoy having more paperclips, that is still extremely bad. The very act of killing us makes these AIs not a worthy successor of the human species.

This suggests that proposing to pause AI today is like proposing to pause electricity in 1880

The prospect of AI killing all of us makes these very different. Yes, in both cases a pause will probably slow GDP growth. But humans should be willing to accept lower GDP if this notably reduces the chance of all humans being killed.

I think a more important reason is the additional value of the information and the option value. It's very likely that the change resulting from AI development will be irreversible. Since we're still able to learn about AI as we study it, taking additional time to think and plan before training the most powerful AI systems seems to reduce the likelihood of being locked into suboptimal outcomes. Increasing the likelihood of achieving "utopia" rather than landing into "mediocrity" by 2 percent seems far more important than speeding up utopia by 10 years.

1. My understanding is that relatively few EAs are actual hardcore classic hedonist utilitarians. I think this is ~sufficient to explain why more haven't become accelerationists.
2. Have you cornered a classic hedonist utilitarian EA and asked them? Have you cornered three? What did they say?

AI x-risk is unique because humans would be replaced by other beings, rather than completely dying out. This means you can't simply apply a naive argument that AI threatens total extinction of value

Paul Christiano wrote a piece a few years ago about ensuring that misaligned ASI is a “good successor” (in the moral value sense),^[1] as a plan B to alignment (Medium version; LW version). I agree it’s odd that there hasn’t been more discussion since.^[2]

Here's a non-exhaustive list of guesses for why I think EAs haven't historically been sympathetic [...]: A belief that AIs won't be conscious, and therefore won't have much moral value compared to humans.

I’ve wondered about this myself. My take is that this area was overlooked a year ago, but there’s now some good work being done. See Jeff Sebo’s Nov ‘23 80k podcast episode, as well as Rob Long’s episode, and the paper that the two of them co-authored at the end of last year: “Moral consideration for AI systems by 2030”. Overall, I’m optimistic about this area becoming a new forefront of EA.

accelerationism would have, at best, temporary effects

I’m confused by this point, and for me this is the overriding crux between my view and yours. Do you really not think accelerationism could have permanent effects, through making AI takeover, or some other irredeemable outcome, more likely?

Are you sure there will ever actually be a "value lock-in event"?

I’m not sure there’ll be a lock-in event, in the way I can’t technically be sure about anything, but such an event seems clearly probable enough that I very much want to avoid taking actions that bring it closer. (Insofar as bringing the event closer raises the chance it goes badly, which I believe to be a likely dynamic. See, for example, the Metaculus question, “How does the level of existential risk posed by AGI depend on its arrival time?”, or discussion of the long reflection.)

1. ^{^}

   Although, Paul’s argument routes through acausal cooperation—see the piece for details—rather than through the ASI being morally valuable in itself. (And perhaps OP means to focus on the latter issue.) In Paul’s words:

   Clarification: Being good vs. wanting good

   We should distinguish two properties an AI might have:

   - Having preferences whose satisfaction we regard as morally desirable.
   - Being a moral patient, e.g. being able to suffer in a morally relevant way.

   These are not the same. They may be related, but they are related in an extremely complex and subtle way. From the perspective of the long-run future, we mostly care about the first property.

2. ^{^}

   There was a little discussion a few months ago, here, but none of what was said built on Paul’s article.

Under purely longtermist views, accelerating AI by 1 year increases available cosmic resources by 1 part in 10 billion. This is tiny. So the first order effects of acceleration are tiny from a longtermist perspective.

Thus, a purely longtermist perspective doesn't care about the direct effects of delay/acceleration and the question would come down to indirect effects.

I can see indirect effects going either way, but delay seems better on current margins (this might depend on how much optimism you have on current AI safety progress, governance/policy progress, and whether you think humanity retaining control relative to AIs is good or bad). All of these topics have been explored and discussed to some extent.

When focusing on the welfare/preferences of currently existing people, I think it's unclear if accelerating AI looks good or bad, it depends on optimism about AI safety, how you trade-off old people versus young people, and death via violence versus death from old age. (Misaligned AI takeover killing lots of people is by no means assured, but seems reasonably likely by default.)

I expect there hasn't been much investigation of accelerating AI to advance the preferences of currently existing people because this exists at a point on the crazy train that very few people are at. See also the curse of cryonics:

the "curse of cryonics" is when a problem is both weird and very important, but it's sitting right next to other weird problems that are even more important, so everyone who's able to notice weird problems works on something else instead.

I generally agree that we should be more concerned about this. In particular, I find people who will happily approve Shut Up and Multiply sentiment but reject this consideration suspect in their reasoning.

A more extreme version of this is that, given the massively greater efficiency with which a digital consciousness could convert matter and energy to utilons (IIRC naively about 3 orders of magnitude according to Bostrom, before any increase from greater coordination), on strict expected value reasoning you have to be extremely confident that this won't happen - or at least have a much stronger rebuttal than 'AI won't necessarily be conscious'.

Separately, I think there might be a case for accelerationism even if you think it increases the risk of AI takeover and that AI takeover is bad, on the grounds that in many scenarios advancing faster might still increase the probability of human descendants getting through the time of perils before some other threat destroys us (every year we remain in our current state is another year in which we run the risk of, for example, a global nuclear war or civilisation-ending pandemic).

(edit: my point is basically the same as emre's)

I think there is very likely at some point going to be some sort of transition to a world where AIs are effectively in control. It seems worth it to slow down on the margin to try to shape this transition as best we can, especially slowing it down as we get closer to AGI and ASI. It would be surprising to me if making the transfer of power more voluntary/careful led to worse outcomes (or only led to slightly better outcomes such that the downsides of slowing down a bit made things worse).

Delaying the arrival of AGI by a few years as we get close to it seems good regardless of parameters like the value of involuntary-AI-disempowerment futures. But delaying the arrival by 100s of years seems more likely bad due to the tradeoff with other risks.

A lot of these points seem like arguments that it's possible that unaligned AI takeover will go well, e.g. there's no reason not to think that AIs are conscious, or will have interesting moral values, or etc.

My stance is that we (more-or-less) know humans are conscious and have moral values that, while they have failed to prevent large amounts of harm, seem to have the potential to be good. AIs may be conscious and may have welfare-promoting values, but we don't know that yet. We should try to better understand whether AIs are worthy successors before transitioning power to them.

Probably a core point of disagreement here is whether, presented with a "random" intelligent actor, we should expect it to promote welfare or prevent suffering "by default". My understanding is that some accelerationists believe that we should. I believe that we shouldn't. Moreover I believe that it's enough to be substantially uncertain about whether this is or isn't the default to want to take a slower and more careful approach.

A presumption in favor of human values over unaligned AI values for some reasons that aren't based on strict impartial utilitarian arguments. These could include the beliefs that: (1) Humans are more likely to have "interesting" values compared to AIs, and (2) Humans are more likely to be motivated by moral arguments than AIs, and are more likely to reach a deliberative equilibrium of something like "ideal moral values" compared to AIs.

I don't think this is a crux. Even if you prefer unaligned AI values over likely human values (weighted by power), you'd probably prefer doing research on further improving AI values over speeding things up.

I think misaligned AI values should be expected to be worse than human values, because it's not clear that misaligned AI systems would care about eg their own welfare.

Inasmuch as we expect misaligned AI systems to be conscious (or whatever we need to care about them) and also to be good at looking after their own interests, I agree that it's not clear from a total utilitarian perspective that the outcome would be bad.

But the "values" of a misaligned AI system could be pretty arbitrary, so I don't think we should expect that.

So I think it's likely you have some very different beliefs from most people/EAs/myself, particularly:

1. Thinking that humans/humanity is bad, and AI is likely to be better
2. Thinking that humanity isn't driven by ideational/moral concerns^[1]
3. That AI is very likely to be conscious, moral (as in, making better moral judgements than humans), and that the current/default trend in the industry is very likely to make them conscious moral agents in a way humans aren't

I don't know if the total utilitarian/accelerationist position in the OP is yours or not. I think Daniel is right that most EAs don't have this position. I think maybe Peter Singer gets closest to this in his interview with Tyler on the 'would you side with the Aliens or not question' here. But the answer to your descriptive question is simply that most EAs don't have the combination of moral and empirical views about the world to make the argument you present valid and sound, so that's why there isn't much talk in EA about naïve accelerationism.

Going off the vibe I get from this view though, I think it's a good heuristic that if your moral view sounds like a movie villain's monologue it might be worth reflecting, and a lot of this post reminded me of the Earth-Trisolaris Organisation from Cixin Liu's Three Body Problem. If someone's honest moral view is "Eliminate human tyranny! The world belongs to Trisolaris AIs!" then I don't know what else there is to do except quote Zvi's phrase "please speak directly into this microphone".

Another big issue I have with this post is that some of the counter-arguments just seem a bit like 'nu-uh', see: 

But why would we assume AIs won't be conscious?

Why would humans be more likely to have "interesting" values than AIs?

But it would also be bad if we all died from old age while waiting for AI, and missed out on all the benefits that AI offers to humans, which is a point in favor of acceleration. Why would this heuristic be weaker?

These (and other examples) are considerations for sure, but they need to be argued for. I don't think they can just be stated and then say "therefore, ACCELERATE!". I agree that AI Safety research needs to be more robust and the philosophical assumptions and views made more explicit, but one could already think of some counters to the questions that you raise, and I'm sure you already have them. For example, you might take a view (ala Peter Godfrey-Smith) that a certain biological substrate is necessary for conscious.

Similarly on total utilitarianism emphasis larger population sizes, agreed to the extent that the greater population increase the population utility, but this is the repugnant conclusion again. There's a stopping point even in that scenario where an ever larger population decreases total utility, which is why in Parfit's scenario it's full of potatoes and muzak rather than humans crammed into battery cages like factory-farmed animals. Empirically, naïve accelerationism may tend toward the latter case in practice, even if there's a theoretical case to be made for it.

There's more I could say, but I don't want to make this reply too long, and I think as Nathan said it's a point worth discussing. Nevertheless it seems our different positions on this are built on some wide, fundamental divisions about reality and morality itself, and I'm not sure how those can be bridged, unless I've wildly misunderstood your position.

1. ^{^}

   this is me-specific

Under preference utilitarianism, it doesn't necessarily matter whether AIs are conscious.

I'm guessing preference utilitarians would typically say that only the preferences of conscious entities matter. I doubt any of them would care about satisfying an electron's "preference" to be near protons rather than ionized.

Strongly there should be more explicit defences of this argument. 

One way of doing this in a co-operative way might working on co-operative AI stuff, since it seems to increase the likelihood that misaligned AI goes well, or at least less badly. 

My personal reason for not digging into this is that my naive model of how good the AI future is: quality_of_future * amount_of_the_stuff. And there is distinction I haven't seen you acknowledged: while high "quality" doesn't require humans to be around, I ultimately judge quality by my values. (Thing being conscious is an example. But this also includes things like not copy-pasting the same thing all over, not wiping out aliens, and presumably many other things I am not aware of. IIRC Yudkowsky talks about cosmopolitanism being a human value.) Because of this, my impression is that if we hand over the future to a random AI, the "quality" will be very low. And so we can currently have a much larger impact by focusing on increasing the quality. Which we can do by delaying "handing over the future to AI" and picking a good AI to hand over to. IE, alignment.

(Still, I agree it would be nice if there was a better analysis of this, which exposed the assumptions.)

Great points, Matthew! I have wondered about this too. Relatedly, readers may want to check the sequence otherness and control in the age of AGI from Joe Carlsmith, in particular, Does AI risk “other” the AIs?.

One potential argument against accelerating AI is that it will increase the chance of catastrophes which will then lead to overregulating AI (e.g. in the same way that nuclear power arguably was overregulated).

Thanks for writing on this important topic!

I think it's interesting to assess how popular or unpopular these views are within the EA community. This year and last year, we asked people in the EA Survey about the extent to which they agreed or disagreed that:

Most expected value in the future comes from digital minds' experiences, or the experiences of other nonbiological entities.

This year about 47% (strongly or somewhat) disagreed, while 22.2% agreed (roughly a 2:1 ratio).

However, among people who rated AI risks a top priority, respondents leaned towards agreement, with 29.6% disagreeing and 36.6% agreeing (a 0.8:1 ratio).^[1]

Similarly, among the most highly engaged EAs, attitudes were roughly evenly split between 33.6% disagreement and 32.7% agreement (1.02:1), with much lower agreement among everyone else.

This suggests to me that the collective opinion of EAs, among those who strongly prioritise AI risks and the most highly engaged is not so hostile to digital minds. Of course, for practical purposes, what matters most might be the attitudes of a small number of decisionmakers, but I think the attitudes of the engaged EAs matters for epistemic reasons. 

1. ^{^}

   Interestingly, among people who merely rated AI risks a near-top priority, attitudes towards digital minds were similar to the sample as a whole. Lower prioritisation of AI risks were associated with yet lower agreement with the digital minds item.

I haven't read your other recent comments on this, but here's a question on the topic of pausing AI progress. (The point I'm making is similar to what Brad West already commented.)

Let's say we grant your assumptions (that AIs will have values that matter the same as or more than human values and that an AI-filled future would be just as or more morally important than one with humans in control). Wouldn't it still make sense to pause AI progress at this important junction to make sure we study what we're doing so we can set up future AIs to do as well as (reasonably) possible?

You say that we shouldn't be confident that AI values will be worse than human values. We can put a pin in that. But values are just one feature here. We should also think about agent psychologies and character traits and infrastructure beneficial for forming peaceful coalitions. On those dimensions, some traits or setups seem (somewhat robustly?) worse than others?

We're growing an alien species that might take over from humans. Even if you think that's possibly okay or good, wouldn't you agree that we can envision factors about how AIs are built/trained and about what sort of world they are placed in that affect whether the future will likely be a lot better or a lot worse?

I'm thinking about things like:

• pro-social insctincts (or at least absence of anti-social ones)
• more general agent character traits that do well/poorly at forming peaceful coalitions
• agent infrastructure to help with coordinating (e.g., having better lie detectors, having a reliable information environment or starting out under the chaos of information warfare, etc.)
• initial strategic setup (being born into AI-vs-AI competition vs. being born in a situation where the first TAI can take to proceed slowly and deliberately)
• maybe: decision-theoretic setup to do well in acausal interactions with other parts of the multiverse (or at least not do particularly poorly)

If (some of) these things are really important, wouldn't it make sense to pause and study this stuff until we know whether some of these traits are tractable to influence?

(And, if we do that, we might as well try to make AIs have the inclination to be nice to humans, because humans already exist, so anything that kills humans who don't want to die frustrates already-existing life goals, which seems worse than frustrating the goals of merely possible beings.)

I know you don't talk about pausing in your above comment -- but I think I vaguely remember you being skeptical of it in other comments. Maybe that was for different reasons, or maybe you just wanted to voice disagreement with the types of arguments people typically give in favor of pausing?

FWIW, I totally agree with the position that we should respect the goals of AIs (assuming they're not just roleplayed stated goals but deeply held ones -- of course, this distinction shouldn't be uncharitably weaponized against AIs ever being considered to have meaningful goals). I'm just concerned because whether the AIs respect ours in turn, especially when they find themselves in a position where they could easily crush us, will probably depend on how we build them.

I think it's interesting and admiral that you're dedicated on a position that's so unusual in this space.

I assume I'm in the majority here that my intuitions are quite different from yours, however. 

One quick point when we're here:
> this view is likely rooted in a bias that automatically favors human beings over artificial entities—thereby sidelining the idea that future AIs might create equal or greater moral value than humans—and treating this alternative perspective with unwarranted skepticism.

I think that a common, but perhaps not well vocalized, utilitarian take is that humans don't have much of a special significance in terms of creating well-being. The main option would be a much more abstract idea, some kind of generalization of hedonium or consequentialism-ium or similar. For now, let's define hedonium as "the ideal way of converting matter and energy into well-being, after a great deal of deliberation." 

As such, it's very tempting to try to separate concerns and have AI tools focus on being great tools, and separately optimize hedonium to be efficient at being well-being. While I'm not sure if AIs would have zero qualia, I'd feel a lot more confident that they will have dramatically less qualia per unit resources than a much more optimized substrate.

If one follows this general logic, then one might assume that it's likely that the vast majority of well-being in the future would exist as hedonium, not within AIs created to ultimately make hedonium.

One less intense formulation would be to have both AIs and humans focus only on making sure we get to the point where we much better understand the situation with qualia and hedonium (a la the Long Reflection), and then re-evaluate. In my strategic thinking around AI I'm not particularly optimizing for the qualia of the humans involved in the AI labs or the relevant governments. Similarly, I'd expect not to optimize hard for the qualia in the early AIs, in the period when we're unsure about qualia and ethics, even if I thought they might have experiences. I would be nervous if I thought this period could involve AIs having intense suffering or be treated in highly immoral ways. 

I think for me, part of the issue with your posts on this (which I think are net positive to be clear, they really push at significant weak points in ideas widely held in the community) is that you seem to be sort of vacillating between three different ideas, in a way that conceal that one of them, taken on its own sounds super-crazy and evil:

1) Actually, if AI development were to literally lead to human extinction, that might be fine, because it might lead to higher utility.

2) We should care about humans harming sentient, human-like AIs as much as we care about AIs harming humans.

3) In practice, the benefits to current people from AI development outweigh the risks, and the only moral views which say that we should ignore this and pause in the face of even tiny risks of extinction from AI because there are way more potential humans in the future, in fact, when taken seriously, imply 1), which nobody believes. 

1) feels extremely bad to me, basically a sort of Nazi-style view on which genocide is fine if the replacing people are superior  utility generators (or I guess, inferior but sufficiently more numerous). 1) plausibly is a consequence of classical utilitarianism (even maybe on some person-affecting versions of classical utilitarianism I think), but I take this to be a reason to reject pure classical utilitarianism, not a reason to endorse 1). 2) and 3), on the other hand, seem reasonable to me. But the thing is that you seem at least sometimes to be taking AI moral patienthood as a reason to push on in the face of uncertainty about whether AI will literally kill everyone. And that seems more like 1) than 2) or 3). 1-style reasoning supports the idea that AI moral patienthood is a reason for pushing on with AI development even in the face of human extinction risk, but as far as I can tell 2) and 3) don't. At the same time though I don't think you mean to endorse 1).