Can you elaborate on what you mean by “the EA-offered money comes with strings?”

But the AIs-with-different-values – even: the cooperative, nice, liberal-norm-abiding ones – might not even be sentient! Rather, they might be mere empty machines. Should you still tolerate/respect/etc them, then?"

The flavor of discussion here on AI sentience that follows what I've quoted above always reminds me of, and I think is remarkably similar to, the content of this scene from the Star Trek: The Next Generation episode "The Measure of a Man." It's a courtroom drama-style scene in which Data, an android, is threatened by a scientists who wants to make copies of Data and argues he's property of the Federation. Patrick Stewart, playing Jean Luc-Picard, defending Data, makes an argument similar to Joe's.

You see, he's met two of your three criteria for sentience, so what if he meets the third. Consciousness in even the smallest degree. What is he then? I don't know. Do you? (to Riker) Do you? (to Phillipa) Do you? Well, that's the question you have to answer. Your Honour, the courtroom is a crucible. In it we burn away irrelevancies until we are left with a pure product, the truth for all time. Now, sooner or later, this man or others like him will succeed in replicating Commander Data. And the decision you reach here today will determine how we will regard this creation of our genius. It will reveal the kind of a people we are, what he is destined to be. It will reach far beyond this courtroom and this one android. It could significantly redefine the boundaries of personal liberty and freedom, expanding them for some, savagely curtailing them for others. Are you prepared to condemn him and all who come after him to servitude and slavery?

This seems awesome! Thanks for sharing.

We’ve tried to do research-based meetings in the past, but we’ve found that people tend to just focus on debating abstract or shallow topics and we haven’t been able to sufficiently incentivize diving in to the more nitty-gritty details or really digging for cruxes. This might not have worked for us because we tried to have too much control over the research process, or because we presented the activity as a debate, or maybe because of the makeup of our group.

Some questions: Did all of the meetings go well? Did you notice any of the issues I mentioned (if not, any idea why)? How many people did you do this with? Were they all post-Intro fellows or selected for in some other way? How much progress did you make on the questions?

Agreed!

I'm not suggesting this in any serious way, and I don't know anything about Bregman or this organization, but an interesting thought comes to mind—I've often heard people ask something along the lines of "should we rebrand EA?" and the answer is "maybe, but that's not probably not feasible." If this organization is truly so good at growth, is based on the same core principles EA is based on (it might not be beyond the shallow "OOO"), and it hasn't been aspersed or tarnished by SBF etc—prima facie it might not be so bad for the EA brand to recede and for currently EA invididuals and institutions to transitions to SMA (SoMA?) ones.

Edit: it's SfMA, I realize now, but I care too much for my bad pun that I'll keep it there...

FWIW, the "deals and fairness agreement" section of this blogpost by Karnofsky seems to agree about (or at least discuss) trade between different worldviews :

It also raises the possibility that such “agents” might make deals or agreements with each other for the sake of mutual benefit and/or fairness.

Methods for coming up with fairness agreements could end up making use of a number of other ideas that have been proposed for making allocations between different agents and/or different incommensurable goods, such as allocating according to minimax relative concession; allocating in order to maximize variance-normalized value; and allocating in a way that tries to account for (and balance out) the allocations of other philanthropists (for example, if we found two worldviews equally appealing but learned that 99% of the world’s philanthropy was effectively using one of them, this would seem to be an argument – which could have a “fairness agreement” flavor – for allocating resources disproportionately to the more “neglected” view). The “total value at stake” idea mentioned above could also be implemented as a form of fairness agreement. We feel quite unsettled in our current take on how best to practically identify deals and “fairness agreements”; we could imagine putting quite a bit more work and discussion into this question.

Different worldviews are discussed as being incommensurable here (under which maximizing expected choice-worthiness doesn't work). My understanding though is that the (somewhat implicit but more reasonable) assumption being made is that under any given worldview, philanthropy in that worldview's preferred cause area will always win out in utility calculations, which makes sort of deals proposed in "A flaw in a simple version of worldview diversification" not possible/useful. 

Some related thoughts and questions:

One day some historian of effective altruism will marvel at how easily it transformed itself.   
—Michael Lewis, Going Infinite

A universal refusal to propagate the human species would be the greatest of conceivable crimes from a Utilitarian point of view
—Henry Sidgwick, The Method of Ethics

NunoSempere points out that EA could have been structured in a radically different way, if the "specific cultural mileu" had been different. But I think this can be taken even further. I think that it's plausible that if a few moments in the history of effective altruism had gone differently, the social makeup—the sort of people that make up the movement—and their axiological worldviews—the sorts of things they value—might have been radically different too. 

As someone interested in the history of ideas, I'm fascinated by what our movement has that made it significantly different than the most likely counterfactual movements. Why is effective altruism the way it is? A number of interesting brief histories have been written about the history of EA (and longer pieces about more specific things like Moynihan's excellent X-Risk) but I often feel that there are a lot of questions about the movement's history, especially regarding tensions that seem to present themselves between the different worldviews that make up EA.

For example,

1. How much was it the individual "leaders" of EA who brought together different groups of people to create a big-tent EA, as opposed to the communities themselves already being connected? (Toby Ord says that he connected the Oxford GWWC/EA community to the rationality community, but people from both of these "camps" seem to be at Felicifia together in the late 2000s.) 
2. When connecting the history of thought, there's a tendency to put thinkers after one another in lineages as if they all read and are responding to those who came before them. Parfit lays the ground for longtermism in the the late 20th century in Reasons and Persons and Bostrom continues the work when presenting the idea of x-risk in 2001. Did Bostrom know of and expand upon Parfit's work, or was Bostrom's framing independent of that, based on risks discussed by the Extropians, Yudkowsky, SL4, etc? There (maybe) seems to be multiple discovery of early EA ideas in separate creation of the Oxford/GWWC community and GiveWell. Is something like that going on for longtermism/x-risk?
3. What would EA look like today without Yudkowsky? Bostrom? Karnofsky/Hassenfeld? MacAskill/Ord? 
4. What would EA look like today without Dustin Moskovitz? Or if we had another major donor? (One with different priorities?)
5. What drove the "longtermist turn?" A shift driven by leaders or by the community?

A few interesting Yudkowsky (not be taken as current opinions, for historical purposes) quotes (see also Extropian Archaeology):

From Eliezer Yudkowsky on the SL4 mailing list, April 30, 2003:

Since the lack of people is a blocker problem, I think I may have to split my attention one more time, hopefully the last, and write something to attract the people we need. My current thought is a book on the underlying theory and specific human practice of rationality, which is something I'd been considering for a while. It has at least three major virtues to recommend it. (1): The Singularity movement is a very precise set of ideas that can be easily and dangerously misinterpreted in any
number of emotionally attractive, rationally repugnant directions, and we need something like an introductory course in rationality for new members.

(2): Only a few people seem to have understood the AI papers already online, and the more recent theory is substantially deeper than what is currently online; I have been considering that I need to go back to the basics in order to convey a real understanding of these topics. Furthermore, much of the theory needed to give a consilient description of rationality is also prerequisite to correctly framing the task of building a seed AI. 

(3): People of the level SIAI needs are almost certainly already rationalists; this is the book they would be interested in. I don't think we'll find the people we need by posting a job opening. Movements often start around books; we don't have our book yet.

It's fascinating to me that this is the reason that there's a "rationality" community around today. (See also) What would EA look like without it? Would it really be any less rational? What does a transhumanisty non-AI-worried EA look like?—I feel like that's what we might have had without Yudkowsky. 

One last thing:

From Eliezer Yudkowsky on the Extropians mailing list, May 12, 2001:

Nick Bostrom wrote:
>
> I now have a presentable version of the paper I presented at the most
> recent Foresight gathering. It's available in two formats:
>
> http://www.nickbostrom.com/existential/risks.html
> http://www.nickbostrom.com/existential/risks.doc
 

I was there for the presentation and I, literally, felt slightly sick to
my stomach. I'd like to endorse "Existential Risks" as being scary and
well worth reading.