A quick comment jumping off the $>20M of evidence collection work...

I've always been curious how much money has been spent to-date on global health research. If we make an (unrealistic) assumption that we could repeat GiveWell for animals, then we'd also have to fund the research that came before. Getting some kind of number, even an ultra-lower bound, would help anchor discussions on evidence-generation in animal welfare.

A starting point could be working backwards from malaria nets. What are the studies that are cited in GiveWell's spreadsheet model for malaria nets? How much did each of those studies cost? How much did it cost to train the researchers who conducted those studies? What's the bottom-line number we get from adding up all this up?

Would love to hear if this exists / see someone research it

Until recently, I always had the impression that there was a glut of animal activists and there'd be little point in me participating. It's not something I ever bothered to check.

For one thing, I heard plenty of stories of how hard it is to get a job at an animal organization. So I figured that would be same for animal activism and that each campaign was saturated with volunteers.

And for another thing, I usually don't hear about pressure campaigns unless they're successful or have tons of people. Understandably nobody wants to promote the mediocre attempts where there's only 3 people pressuring a target and failing. Activist campaigns want to seem like a cool group on the cusp of victory while newspapers have better stories to run. 

It wasn't until I said all this to an organizer a few weeks ago that I learned all the struggles with recruiting and retaining activists. And then I made a mental note to join the next action I could. 

I don't know how many people would benefit from hearing, "Hey our capacity is actually very low and we'd benefit from any non-crazy person with a pulse", but I certaintly did. And now I'm among the irregular participants who show up at least once or twice a year.

Don’t have answers but just wanted to say I really appreciate this mini-compilation of what’s already been done 

For what it's worth, I read that abstract as saying something like, "within the class of interventions studied so far, the literature has yet to settle onto any intervention that can reliably reduce animal product consumption by a meaningful amount, where meaningful amount might be a 1% reduction at Costco scale or long-term 10% reduction at a single cafeteria. The class of interventions being studied tends to be informational and nudge-style interventions like advertising, menu design, and media pamphlets. When effect sizes differ for a given type of intervention, the literature has not offered a convincing reason why a menu-design choice works in one setting versus another."

Okay, now that I've typed that up, I can see why "unsolved problem" is unclear. 

And I'm probably taking a lot of leaps of faith in interpretation here

Seth, for what it's worth, I found your hourly estimates (provided in these forum comments but not something I saw in the evaluator response) on how long the extensions would take to be illuminating. Very rough numbers like this meta-analysis taking 1000 hours for you or a robustness check taking dozens / hundreds of hours more to do properly helps contextualize how reasonable the critiques are.

It's easy for me (even now while pursuing research, but especially before when I was merely consuming it) to think these changes would take a few days. 

It's also gives me insight into the research production process. How long does it take to do a meta-analysis? How much does rigor cost? How much insight does rigor buy? What insight is possible given current studies? Questions like that help me figure out whether a project is worth pursuing and whether it's compatible with career incentives or more of a non-promotable task

Would you recommend that I share any such posts with both the authors and the evaluators before making them?

Yes. But zooming back out, I don't know if these EA Forum posts are necessary.

A practice I saw i4replication (or some other replication lab) is that the editors didn't provide any "value-added" commentary on any given paper. At least, I didn't see these in any tweets they did. They link to the evaluation reports + a response from the author and then leave it at that.

Once in a while, there will be a retrospective on how the replications are going as a whole. But I think they refrain from commenting on any paper.

If I had to rationalize why they did that, my guess is that replications are already an opt-in thing with lots of downside. And psychologically, editor commentary has a lot more potential for unpleasantness. Peer review tends to be anonymous so it doesn't feel as personal because the critics are kept secret. But editor commentary isn't secret...actually feels personal, and editors tend to have more clout.

Basically, I think the bar for an editor commentary post like this should be even higher than the usual process. And the usual evaluation process already allows for author review and response. So I think a "value-added" post like this should pass a higher bar of diplomacy and insight.

Chiming in here with my outsider impressions on how fair the process seems

@david_reinstein If I were to rank the evaluator reports, evaluation summary, and the EA Forum post in which ones seemed the most fair, I would have ranked the Forum post last. It wasn't until I clicked through to the evaluation reports that I felt the process wasn't so cutting.

Let me focus on one very specific framing in the Forum post, since it feels representative. One heading includes the phrase "this meta-analysis is not rigorous enough". This has a few connotations that you probably didn't mean. One, this meta-analysis is much worse than others. Two, the claims are questionable. Three, there's a universally correct level of quality that meta-analyses should reach and anything that falls short of that is inadmissible as evidence. 

In reality, it seems this meta-analysis is par for the course in terms of quality. And it was probably more difficult to do so given the heterogeneity in the literature. And the central claim of the meta-analysis doesn't seem like something either evaluator disputed (though one evaluator was hesitant). 

Again, I know that's not what you meant and there are many caveats throughout the post. But it's one of a few editorial choices that make the Forum post seem much more critical than the evaluation reports, which is a bit unusual since the Evaluators are the ones who are actually critiquing the paper.

Finally, one piece of context that felt odd not to mention was the fundamental difficulty of finding an expert in both food consumption and meta-analysis. That limits the ability of any reviewer to make a fair evaluation. This is acknowledged at the bottom of the Evaluation Summary. Elsewhere, I'm not sure where it's said. Without that mentioned, I think it's easy for a casual reader to leave thinking the two Evaluators are the "most correct".

Really enjoyed this. Not much public debate in this space as far as I can see. To two of your cruxes:

Is meta-analysis even useful in these contexts, with heterogeneous interventions, outcomes, and analytical approaches?

Will anyone actually do/fund/reward rigorous continued work?

I've sometimes wondered if it'd be worth funding a "mega study" like Milkman et al. (2021). They tested 54 different interventions to boost exercise among 61,000 members. Something similar for meat reduction could allow for some clean apples-to-apples comparisons.

I've seen the number $2.6 million floating around for how much this intervention costs. Granted, that's probably on top of convincing the mega-team of researchers to work on the project, which might only happen through the prestige of an academic lab. But it's also not an astronomical cost. And there'd be still some learning value from a smaller set of interventions and a smaller sample.  

This might be a better use of resources than striving for the "ideal" meta-analysis, since that sounds expensive too.

Ooh this is neat.

I like how it neutralizes the certainty-seeking part of me since it's only me, the difference maker, that has the option of a guaranteed 100% outcome. For the beneficiary, it's always a gamble.