But then isn't claiming that number simply misleading marketing, or do you have data, that the coaching helped them (how), and that's how you arrived at the number? I feel this really depends on the quality of your survey. I can imagine people who paid for your service may just feel obligated/ or may just want to rationalize their spending from the past by saying, "this helped me in my future dating." "Intentionality in dating" seems very broad; it's worth explaining what you meant :) 

Vasco, let me try to say more clearly what I'm getting at:

Societies have repeatedly let large groups oppress smaller ones because it suited their preferences: white people enslaving Black people, most humans keeping animals in factory farms, religious majorities restricting women's lives. In each case, a small group of activists started by saying: this is wrong, it causes enormous harm, and your preferences don't justify it. And we now think they were right, not just that their preferences eventually spread.

As I understand your view, something is good if you prefer it and don't expect that preference to change on reflection. That makes "good" relative to each person. So imagine someone who says: "I like eating chickens from factory farms, and I fully expect to keep liking it." By your definition, factory farming is good for them and bad for you, and that's the end of the discussion. Why should they care about your preference, any more than you should care about theirs? But in real life, that's not what we do. We say: you're wrong, these chickens suffer enormously, and you should help us end this. I don't see how your view lets us say that and mean it.

I'm also unsure how this works at the level of a society. You say you want to satisfy the preferences of moral agents, all else equal. If 10,000 people stably prefer to keep 100 people enslaved, the preference satisfaction seems to be heavily on the enslavers' side. You said long-term outcomes don't have to be good, but on your view that only means they don't have to be good by your lights. What would make the majority wrong, rather than just different from you?

And a related question: chickens and shrimp aren't moral agents. Do their interests count in themselves on your view, or only because moral agents like you happen to care about them?

I think I'm still missing what kind of system you're trying to build, so I'd really like to understand how you'd answer these. But maybe this is to much to continue on the forum.

Or they can develop a preference to keep slaves, and keep animals in factory farms, because these preferences were/are shared by so many, how will you defend that they're unethical? Let's say you have 100 people enslaved, and 10,000 who think it's a brilliant idea — in your theory, do the strongest preferences win? Because this would quickly become gamified if a large number of people were oppressing a small group (because they don't include this group in their moral circle). 
"They would not be making a mistake from their perspective, in the sense there would not be "an action, decision, or judgment that produces an unwanted or unintentional result" from their perspective." - this is exactly what I mean above: you reduced morality to perspective, so in a society you would just accumulate those and assume whatever this accumulation arrived at is good? Cause meta-ethical views do spill over into practical ethics, and this is then a problem: under some theories, we end up in "do whatever," and under others we end up in "there is something that makes us know that action X or Y is wrong/not desirable/should be outlawed". I can't accept a world where it all depends on the sum of individual preferences; we would make no moral progress if that were the case, and we're clearly expanding the moral circle. 

I wonder whether we have a semantic problem here, or if you actually defend subjectivism.

You say "bad" can only mean "our feelings about it." If badness is our attitude toward pain, then pain is bad because we dislike it, not because of the pain itself. If badness is in the experience of the one suffering, then it's bad whether or not any observer cares, which is all I meant by "bad regardless of whether anyone cares about it."

Very few people care about soil arthropods. If badness were grounded in what humans or moral agents feel about it, their pain would matter little, and estimating it would mostly measure human sentiment. Your thinking it's worth estimating suggests their pain matters independently of anyone caring.

The baby example points the same way. A hungry baby shows hunger is bad for the baby, with no beliefs or attitudes required. What preferences don't give us is impartiality. If preferences are the foundation, nothing obliges a human to weigh the suffering of a shrimp or a mite. Impartiality is the part that needs a justification beyond what anyone happens to feel.

On "what impartial agents would endorse on reflection": do they endorse reducing suffering because it's bad, or is it bad because they endorse it? If the first, the endorsement tracks something prior. If the second, a reflective agent who came to endorse cruelty wouldn't be making any mistake, which I find hard to accept.

I often find that nihilists drop words like "wrong" and "bad" while keeping the activity those words were doing, and that activity usually involves a kind of impartiality nihilism can't account for. Like you treat the question of soil anthropods and their welfare as something one can get right or wrong (not a matter of taste), this to me does not seem preference driven. As de Lazari-Radek and Singer argued, evolution explains partial concern for ourselves and our kin very well, but not concern for creatures that give us nothing back. Caring about them looks less like a built-in preference and more like reason taking us past our preferences.

But yeah I would need more definitions to do this discussion well, just to insure we don't talk pass each other, but I probably can't also engage endlessly on this, as these are unsolvable ethical questions with all pathways being prone to problems.

Just sharing some intuitions on this further, no need to reply.
Given that donors potentially are accounting for short AI timelines, should the money not focus on things like regulatory approvals, building consumer acceptance, and better facilities? As money spent there now will buy time, while the science part will be solved simply by the AI progress and TAI. Especially since you're pointing out to an example, where the main constraint is capital and market access rather than solving unknowns? My intuition also is that a well-designed market test would be something people want to see. Where I would put my effort in terms of pitching this, if I were GFI, is probably a short list of specific bets, their cost estimates, milestones, and kill criteria + making the case for the things AI can't speed up and that you work on already. Or, if you really want to speed up tractable things now, what are the very specific things: cell lines, media optimization, fat alternatives, what exactly needs to be done there that AI labs could help with, and why now vs. waiting for TAI to solve it in 5 -10 years.
 

If I understand this properly from your summary, Vasco (sorry, no time to read the long article), Tomasik lands in something like subjectivism/expressivism? I think naturalism might be a better way out, and it fits physicalist monism well. In naturalism, factory farming/ slavery are bad because of the pain itself, as a natural state in the world, not because of our feelings about it. It's bad regardless of whether anyone cares about it. Obviously, naturalism does not explain why we should care; that's why, e.g., Singer moved towards the "point of view of the universe," with impartiality as an objective truth. I wonder if, by understanding it as a constitutive norm of reasoning itself, we could somehow stay naturalists without the objections that subjectivism faces. I do think most of us think that the suffering is bad for the subject and not because we perceive it as bad, but that's obviously not tackling the part about "what is it that binds us". I feel like morality has an almost Kantian feeling of authority, and I'm unsure we can capture it without stepping outside this monism trip, which might be why Singer is where he is.

Subjectivism is the easiest way out, but it will quickly leave us empty-handed in ethics; we won't be able to defend anything, as there can never be a mistake. But yeah, either we will remain skeptical forever or pick something up. Personally, my brain pushes me into moral uncertainty/philosophical skepticism, but I know a lot of us, understandably, need to anchor in something, sometimes something greater than ourselves, or, in the case above, in part of ourselves :) Sometimes it works, sometimes it doesn't. 

I fixed the number in the comment above, yes, 3.6 to 29. 

Updating on one comment from Siobhan or me does not seem like a good idea :) Give it a go; I think a lot of EAs are just already on the side of your argument here; that's why I thought maybe it's not so much needed for us, as we're already convinced. But publishing your thoughts is always good, and if someone isn't interested, then they'll just skip it. 
The framing might be new, but the problem is old, and you've talked about it for years just in various different framings. At least that's how it sounds to me. But you should not update about posting because 2 people don't find the thoughts original. I think your thoughts on this are needed, and I expressed this above. 

Thanks for this it’s a really useful point and one we’ve been thinking about too. You’re right that the value of cross-cause prioritization research depends a lot on who’s actually listening and what they’re open to. The way we see it, there are at least three types of donors this work is relevant to.

The first are donors who are genuinely cause-neutral and are actively looking for guidance on how to split their resources. Some have asked us for exactly this kind of analysis, both in the past and in the current wave of new philanthropy, so this work is directly for them.

The second are donors who are broadly cause-neutral but, in practice, hedge by putting some money into one cause as a kind of safety net (GHD is a common choice), or who defer to advice that isn’t explicitly modeled. We think an explicit model can help these donors do what they already want to do: diversify their portfolio but in a more principled way. We’re hoping to address their concerns more directly in future posts in the series.

The third group is donors who mostly default to giving later, or to giving within a single cause. Even here, we think the work has something to offer. Many donors with a clear worldview still aren’t completely certain of it, and it’s worth asking what they should do as a hedge against being wrong. Relatedly, some donors who appear to hold a particular worldview may actually be people who think the arguments come down squarely in favor of giving, as though that worldview were true. For them, it can be useful to see an alternative way of allocating resources and what it would recommend. We probably won’t change many minds in this group, but we plan to engage with some of the assumptions behind these positions in later posts.

We’re already in contact with some donors in each of these groups, and we’d love more opportunities to engage with others. Part of our hope for this series is simply that it helps the work get noticed by the people who might find it useful. Thanks again for raising this.

Oh, so you think this is just not adding new perspective to some recent articles? I might have not being following the discussion so closely. So maybe you're right in this context.