Recommendations
-
Evidence does not support the use of informational documentaries.
-
Interventions that change intended eating habits may not result in actual change in eating habits. In our study, even very large changes in intention did not change actual behavior.
-
Future studies should ensure that participants are unaware of the study's purpose. They can do this with blinding.
Key Findings
-
Tested a 20-minute documentary "Good For Us" that highlights the environmental, human health and animal welfare harms of eating meat and other animal products.
-
In a randomized controlled experiment, compared the documentary to a control video (a generic motivational speech). Participants were from the general population of the United States.
-
Followed up 12 days later with a survey that was described as a different study. This helped to "blind" participants to the purpose of the study when collecting data, reducing potential bias.
-
-
Our first study found the documentary had no effect on a variety of different outcomes.
-
Found no reduction in animal product consumption. The average change we measured in one study was less than a 1-ounce reduction in animal product consumption per week, with a 95% confidence interval ranging from a 6 ounce reduction to a 5 ounce increase.
-
Found no change in moral valuation of animals ("speciesism").
-
Found no meaningful increases in interest in animal activism or in perceived importance of environmental sustainability, animal welfare, or eating a healthful diet.
-
-
Our second study was deliberately designed less rigorously, to resemble previous studies that measured intended behavior. Immediately after the documentary, asked viewers if they planned to eat more or less animal products next week. Many viewers planned to eat less animal products in the week after seeing the documentary.
-
Documentary made viewers 242% more likely to intend to reduce meat consumption than participants who viewed the control video. Critically, our first study suggested that these intentions do not actually translate to reductions in consumption.
-
A previous meta-analysis suggested that comparable interventions make people about 22% more likely to intend to reduce their meat consumption. So our documentary was likely very convincing relative to other interventions, but still not effective at reducing consumption in our more rigorously designed study.
-
-
Our third study tried to make the documentary more effective. Added a pledge, goal-setting exercises, and reminder email; and participants were people interested in nutrition research. The documentary still was not effective by any measurement. Also looked at just people who attended at least a 2-year college and identified as Democrats, but still no effect.
Thanks for including the CI bounds, that makes it much more interpretable as an 'actual tight null' rather than an underpowered study.
CF I think that is relative to an average US consumption of about 80 oz per week ... so under +/- 10% with 95% CIs ... and the 80% CI would be obviously even tighter
(Or am I wrong, maybe this isn't that tight)
Disclaimer: Jacob and I both work at Rethink Priorities
Do you think your study is sufficiently well powered to detect very small effect sizes on meat consumption? It seems plausible that effects on meat consumption would be very small in expectation plus many people would not reduce meat no matter what, so you may be needing to detect a small shift within a small subpoulation.
It looks like you have 80% power to detect an effect size of d=0.24 - which is actually substantially larger than the effects we usually find for animal interventions even on more moveable things like attitudes/signing a petition/agreeing that "factory farms aren't great". Their null result on effect on meat consumption was not at all tightly bounded: -0.3oz [-6.12oz to + 5.46oz]
So I think the different results here seem possibly explained just by the fact that you could find effects on the moveable attitudes but were underpowered to detect differences in meat consumption. I'd be curious to estimate what effect size would we be looking at if say 3-5% of people stopped eating meat (an optimistic estimate IMO).
This is perhaps further confounded by a large amount of probable noise - how good are people at estimates oz of meat eaten in different time periods, is oz of meat something that is distributed in a way to corresponds to what a t-test is assessing?
Thanks for these Peter! (Note that Peter and I both work at Rethink Priorities.)
No, and this is by design as you point out. We did try to recruit a population that may be more predisposed to change in Study 3 and looked at even more predisposed subgroups.
I think we were informed by the results of our meta-analysis, which generally found effects around this size for meat reduction interventions.
Obviously, this is ultimately subjective, but this corresponds to plus or minus a burger per week, which seems reasonably precise to me. The standardized CI is [−0.17, 0.15], so bounded below a 'small effect'. And, as David points out, less stringent CIs would look even better. But to be clear, I don't have a substantive disagreement here—just a matter of interpretation.
For even more power, we could combine studies 1 & 3 in a meta-analysis (doubling the sample size). Study 3 found a treatment effect of−1.72 oz/week; 95% CI: [−8.84,5.41], so the meta-analytic estimate would probably be very small but still in the correct direction, with tighter bounds of course.
Just to clarify, we measured attitudes in all 3 studies. We found an effect on intentions in Study 2 where there wasn't blinding and follow-up was immediate. Studies 3 & 4 (likely) didn't find effects on attitudes.
Just roughly taking David Reinstein's number of 80 oz per week (could use our control group's mean for a better estimate) and assuming no other changes, 1% abstention would give a 0.8 oz effect size and 5% 4 oz. So definitely under-powered for the low end, but potentially closer to detectable at the high end. (And keeping in mind this is at 12-day follow-up; we should expect that 1% to dwindle further at longer follow-up. With figures this low I would be pessimistic for the overall impact. But keep in mind other successful meat reduction interventions don't seem to have worked mostly through a few individuals totally abstaining!)
I wouldn't expect issues in testing the difference in means given our samples sizes. But otherwise not sure what you're suggesting here.
This is really interesting, thanks for this! In particular, it was really helpful comparing it to previous less-rigorously designed surveys, as I'm sure you expected pushback using those results. I had a few quite preliminary questions:
Thank you taking the time to engage, much appreciated! Forgive my responding quickly and feel free to ask for clarification if I miss anything:
This might be a stupid question, but do you know if anyone looked at how many new activists can a documentary make? :) I met some people who were animal activists because they watched some inspiring movies about animal suffering. I wonder if despite the fact that the movies don't work as a thing that makes people change their diet, they might make some very specific people into animal activists? :) Probably the cost of producing them would not justify the results of getting like 100 activists out of one movie. But on the other hand, if the activist will later work in an organization that leads to a major policy change in a country, maybe it is worth it? :) Just some fun speculations about flow-through effects :) I wonder also about things like effects on the press coverage of animal-related topics, effects on influencers and celebrities, effects on the perception of animal suffering (e.g. would there more likely vote on a law that e.g. forbade using crates in farming systems, or more likely sign a petition to the government). We all know by now that habits are very hard to change, but opening minds to a different viewpoint - maybe not as hard.
Yes, we did and found no meaningful increases in interest in animal activism, including voting intentions. Full questions available in in the supplementary materials.