An Effective Altruist Message Test

I decided to run an Effective Altruist message on a full population survey I have access to, use bayesian message testing software to analyze the results, and share the results with the EA community on the forum.

I tested several EA themed messages aimed at increasing respondents’ interest in donating and effective altruism. For non-control respondents, I presented them with either the claim that AMF can save a life for 3,500 dollars (Facts), a short version of Peter Singer’s Pond analogy (Obligation) or a short take on Will MacAskill’s opportunity framing (Opportunity).

I then asked respondents how much they planned to donate in the next 12 months, their interest in EA and gave them the opportunity to click on a link to donate to the Against Malaria Foundation (AMF) or potentially join Giving What We Can (GWWC). I recorded whether they clicked on these links. Additionally, I asked how much they donated in the last 12 months as a “prescreen” to control for in my analysis, dramatically increasing my precision. The wording of each of the questions described above is in a doc linked to at the bottom of this article.

Overall, my results indicate that these brief messages do not increase respondents perspective donations, but there is some evidence that the facts and obligation message may increase interest in EA among educated individuals. Below, I discuss my results in more detail.

Methods

This was embedded in a 1200 person online survey representative of US citizens. Within this survey, I delivered 3 treatments (Facts, Obligation and Opportunity) each with a sample size of ~200 respondents. I compared how interested respondents  were in learning more about effective altruism and how much they planned to donate. I examined these relationships overall and among the critical subgroup of those with at least a bachelor’s degree (this was my only pre-planned comparison; having a bachelor’s degree serves as a rough proxy for the EA target elite audience). Unfortunately, only 10 respondents took substantive action by clicking on the AMF donation or GWWC links, so this sample size was not sufficient to conduct robust analysis on this dependent variable.

To analyze this relationship, I used Bayesian hierarchical modeling software based in R and Stan. This modeling approach allows us to create a probability distribution of the treatment effects, reduce variance by controlling for other variables in the survey and borrow power from the full sample when estimating effects among subgroups.

Future Donation Plans

To gauge how the EA messages changed people’s donations, I predicted the probability that respondents plan to donate at least 6% of their income to charity. Below are treatment effects and probability distributions for the effect of each message. Error bars are one standard error.

As evidenced by the plots above, none of the treatments had a large impact. In fact, the impact such as it is is negative. The impact is minute, with Average Treatment Effects (ATEs) well under 1%. Even in the most optimistic case, few are likely to increase their expected donation behaviors in response to the arguments provided. This is not surprising; it can take a lot for people to change substantial behaviors, or even suggest they may change them in a survey. There’s little evidence that any argument is more effective, and will require other means to assess.

Interest in Effective Altruism

I also examined interest in Effective Altruism (whether respondents are at least somewhat interested in learning more about EA). Below are treatment effects and probability distributions for the effect of each message.

Unlike perspective donations, interest exhibits substantial variation between treatments. While the opportunity message appears to actively put people off, letting people know they can save lives for $3,500 does appear to get folks interested in effective altruism, and may be a good way to get them in the door. However, the most interesting ATEs appear when breaking the results out by education level. Below are ATEs, probabilities that each message is best, and probabilities of backlash (a negative effect) for each message among those with at least a bachelor’s degree.

There’s a large gap in the impact these messages have on those with and without a bachelor’s degree. For those with a bachelor’s degree, every message has a positive ATE, and the effects of obligation and facts are quite substantial (they increase the chance individuals are at least somewhat interested in learning more about EA by over 6 percentage points). However, there’s a substantial amount of variance in these effect sizes. The true effect of these interventions could credibly range from slightly negative to over 15 percentage points, but most of the probability mass lies in the middle of this range. Overall, Singer’s pond analogy and the scope of impact people can have with their donations appear to be effective methods of building interest in EA; there’s around a 90% chance that one of these is the best message (out of the three and the control) that you can use with individuals having at least a bachelor’s degree.

However, the results are much different for those without a bachelor's degree. Only the factual argument appears to have any positive effect, and they appear to be turned off by the opportunity and obligation framings. This confirms what we already suspected, prospective EAs are likely to be educated elites and it makes sense to target them.

Discussion and caveats

Overall, these findings are mixed, but not surprising. EA messaging does work to get people interested in Effective Altruism. However, EA messaging alone is not enough to get people to even claim that they will increase their donations. This likely takes a much more substantial treatment. But getting educated elites interested in Effective Altruism is the first step. By emphasizing the moral reasoning behind effective altruism and the scale of good we can do in the world, we can encourage people to learn more about Effective Altruism. From there, we can change their behavior.

Like all research, this is limited. For one, the individuals targeted and the context are not entirely typical. EA messaging will tend to come from friends and acquaintances in person or in discussions online rather than as an anonymous message in a web survey. People may react differently in other situations, but this study does provide an important piece of experimental evidence that can inform how we try to engage people. Additionally, having a bachelor’s degree is not enough to be in EA’s core audience. EAs as a whole tend to be analytically oriented and are often mathematical in their thinking. This population is not as restrictive as typical perspective EAs. The good news is that both of these differences suggest the true effect size may be larger. A more personal contact to an even better target may be very effective at encouraging people to join EA. Indeed, that may help explain EA’s substantial growth. However, in another way, the audience is overly restrictive; only US citizens were included. Different messages may be more effective in other countries.

This study is a step forward, it provides some evidence on what treatments work best and what we can accomplish with a contact. As always, more research (both observational and experimental) is needed. As our community engages in more trials to test our messaging, we can continue to fine tune it and expand the appeal of EA.

Thanks to Kerry Vaughan for advice on message choice.

Question Wording