Recommendations
-
Evidence does not support the use of informational documentaries.
-
Interventions that change intended eating habits may not result in actual change in eating habits. In our study, even very large changes in intention did not change actual behavior.
-
Future studies should ensure that participants are unaware of the study's purpose. They can do this with blinding.
Key Findings
-
Tested a 20-minute documentary "Good For Us" that highlights the environmental, human health and animal welfare harms of eating meat and other animal products.
-
In a randomized controlled experiment, compared the documentary to a control video (a generic motivational speech). Participants were from the general population of the United States.
-
Followed up 12 days later with a survey that was described as a different study. This helped to "blind" participants to the purpose of the study when collecting data, reducing potential bias.
-
-
Our first study found the documentary had no effect on a variety of different outcomes.
-
Found no reduction in animal product consumption. The average change we measured in one study was less than a 1-ounce reduction in animal product consumption per week, with a 95% confidence interval ranging from a 6 ounce reduction to a 5 ounce increase.
-
Found no change in moral valuation of animals ("speciesism").
-
Found no meaningful increases in interest in animal activism or in perceived importance of environmental sustainability, animal welfare, or eating a healthful diet.
-
-
Our second study was deliberately designed less rigorously, to resemble previous studies that measured intended behavior. Immediately after the documentary, asked viewers if they planned to eat more or less animal products next week. Many viewers planned to eat less animal products in the week after seeing the documentary.
-
Documentary made viewers 242% more likely to intend to reduce meat consumption than participants who viewed the control video. Critically, our first study suggested that these intentions do not actually translate to reductions in consumption.
-
A previous meta-analysis suggested that comparable interventions make people about 22% more likely to intend to reduce their meat consumption. So our documentary was likely very convincing relative to other interventions, but still not effective at reducing consumption in our more rigorously designed study.
-
-
Our third study tried to make the documentary more effective. Added a pledge, goal-setting exercises, and reminder email; and participants were people interested in nutrition research. The documentary still was not effective by any measurement. Also looked at just people who attended at least a 2-year college and identified as Democrats, but still no effect.
Thanks for these Peter! (Note that Peter and I both work at Rethink Priorities.)
No, and this is by design as you point out. We did try to recruit a population that may be more predisposed to change in Study 3 and looked at even more predisposed subgroups.
I think we were informed by the results of our meta-analysis, which generally found effects around this size for meat reduction interventions.
Obviously, this is ultimately subjective, but this corresponds to plus or minus a burger per week, which seems reasonably precise to me. The standardized CI is [−0.17, 0.15], so bounded below a 'small effect'. And, as David points out, less stringent CIs would look even better. But to be clear, I don't have a substantive disagreement here—just a matter of interpretation.
For even more power, we could combine studies 1 & 3 in a meta-analysis (doubling the sample size). Study 3 found a treatment effect of−1.72 oz/week; 95% CI: [−8.84,5.41], so the meta-analytic estimate would probably be very small but still in the correct direction, with tighter bounds of course.
Just to clarify, we measured attitudes in all 3 studies. We found an effect on intentions in Study 2 where there wasn't blinding and follow-up was immediate. Studies 3 & 4 (likely) didn't find effects on attitudes.
Just roughly taking David Reinstein's number of 80 oz per week (could use our control group's mean for a better estimate) and assuming no other changes, 1% abstention would give a 0.8 oz effect size and 5% 4 oz. So definitely under-powered for the low end, but potentially closer to detectable at the high end. (And keeping in mind this is at 12-day follow-up; we should expect that 1% to dwindle further at longer follow-up. With figures this low I would be pessimistic for the overall impact. But keep in mind other successful meat reduction interventions don't seem to have worked mostly through a few individuals totally abstaining!)
I wouldn't expect issues in testing the difference in means given our samples sizes. But otherwise not sure what you're suggesting here.