Comments
Excellent steps towards solving some tricky questions. Well done team- looking forward to seeing your reflections on this work progress.
Excellent steps towards solving some tricky questions. Well done team- looking forward to seeing your reflections on this work progress.
We are also looking for comments, feedback and questions on the Quarto 'dynamic document' report that we link incessantly above . You can leave comments on that report here, or in that Quarto itself using the embedded hypothes.is tools (you need to sign up for an account with them, , which takes about 30 seconds.)
The trial was run in conjunction with Josh Lewis (NYU). Thanks to David Moss and others for feedback on this post, and to Jamie Elsey for support with the Bayesian analysis.
Giving What We Can together with the EA Market Testing Team (EAMT) tested marketing and messaging themes on Facebook in their Effective Giving Guide Facebook Lead campaigns which ran from late November 2021 - January 2022. GWWC's Giving Guide answers key questions about effective giving and includes the latest effective giving recommendations to teach donors how to do the most good with their donations. These were exploratory trials to identify promising strategies to recruit people for GWWC and engage people with EA more broadly.[1] We report the most interesting patterns from these trials to provide insight into which hypotheses might be worth exploring more rigorously in future (‘confirmatory analysis’) work.
Across four trials we compared the effectiveness of different types of (1) messages, (2) videos, and (3) targeted audiences. The key outcomes were (i) email addresses per dollar (when a Facebook user provides an email lead) and (ii) link clicks per dollar. Based on our analysis of 682,577 unique Facebook ‘impressions’, we found:[2]
One of the primary goals of the EAMT is to identify the most effective, scalable strategies for marketing EA. Our main approach is to test marketing and messaging themes in naturally-occurring settings (such as advertising campaigns on Facebook, YouTube, etc.), targeting large audiences, to determine which specific strategies work best in the most relevant contexts. In this report, we share key patterns and insights about the effectiveness of different marketing and messaging approaches used in GWWC's Effective Giving Guide Facebook Lead campaigns. The patterns we share here serve as a starting point to consider themes and hypotheses to test more rigorously in our ongoing research project.
We are hoping for feedback and suggestions from the EA community on these trials and their implementation and analysis. We continue to conduct detailed analyses of this data.[6] We'd like to get ideas from the community about how to improve trials like these and what other analyses would be informative. Additionally, we hope this report will give other groups ideas for messaging themes to try and tools and processes for starting and analyzing marketing campaigns.[7] Finally, we seek to keep the community abreast of what we (at EA Market Testing and at GWWC) are up to (see the EA Market Testing Gitbook page for more details and resources).
In these trials, we aimed to test two different approaches to messaging– (1) presenting facts about effective giving and (2) presenting cause-focused messages–in order to get people to provide their email to download GWWC's Giving Guide, which also subscribed them to GWWC’s email list. We tested (i) six short animated videos with facts about effective giving or a focus on specific cause areas and (ii) seven messages displayed with the videos on (iii) different segments: a general Facebook audience, audiences based on specific interests (animal rights, climate change, poverty, philanthropy), and a lookalike audience (people with a similar profile as current people engaging with GWWC).
There were two dimensions of treatment content: (1) the text displayed above the videos and (2) the video ad's theme and content.
Facts about effective giving
Cause-focus
Arguments, rich content from brand video
Outcome measures used in our analysis of the messages and videos were (1) email addresses per dollar and (2) link clicks per dollar. When we say 'results' below, we refer to these outcome measures. Other measures collected were amount spent, cost per impression, cost per link click, link click-through rate, and 3-second video plays.
The cost was determined in the Facebook ad auction for each impression on a per-result basis. However, the ‘cost-per-result' is determined in part by factoring in the likelihood of getting the result. Results cost less if you are expected to get more results, so ultimately, marketers are paying for the value of impressions.[8] Generally, when we talk about a segment being 'expensive' we mean that it’s expensive to serve them an impression - not that it’s necessarily expensive on a cost-per-result basis.
Facebook allows you to generate pivot tables showing the number of results for different segments. We used these pivot tables to generate datasets with a row for each impression and downloaded these into our repo for further analysis.[9] We chose to treat each unique user impression (aka 'reach') as an independent observation and each result or link click as if it came from a different 'reach.' This enabled us to do more sophisticated statistical analyses than those available on the Facebook ads platform. Still, our approach has its limitations. Facebook only gives the total number of results of all people in a segment who saw an ad, so we do not know if the same person contributes more than one result to the total. (However, in this context it seems unlikely that many people would give their email to sign up for giving guides more than once.)
Divergent delivery: Facebook does not facilitate randomized (or ‘balanced’) assignment. Unlike in standard experiments and RCTs, Facebook ads do not ensure that each participant has the same chance of ending up in each condition, nor does it allow an easy way to re-weight for imbalance. Specifically, Facebook’s algorithm will always try to assign a particular content variation to people it thinks will be most receptive to that specific variant. All our results may come from…
Thus, given these idiosyncrasies, our results may not generalize much beyond Facebook, nor even to an evolving Facebook environment. (We discuss this more fully in our Gitbook here, where we maintain and update a knowledge base on these design and implementation issues.)
All analyses were conducted on an exploratory, non-preregistered basis. We had no prior hypotheses before conducting the research. The results we are highlighting were not findings predicted in advance. We made many comparisons between different segments, most of which did not show any substantial interesting difference. Although we did conduct statistical tests, we don’t report them in detail here we do this for brevity and because of the limited potential for generalizable causal inference, given the caveats above. We will report a richer statistical analysis and summary, along with a complete pipeline of code, data, and results, in the transparent ‘Quarto’ dynamic document HERE, where we are working on a Bayesian 'decision relevant’ approach. (See, e.g., plots here and here.) More information about treatment assignment by campaign is explained in the Appendix to this post.
Note: We include charts that depict ‘results’ (link clicks or emails) as well as ‘results per dollar.’ [10]
We defined the following audiences. Within each of these groups, Facebook chose which ad content to allocate according to its maximizing algorithm.[12]
Results
When ‘Climate’ and ‘Global Poverty’ audiences were presented videos ‘for their own causes’, they performed worse than the ‘Philanthropy’ audience when presented these same videos.[15]
Aggregated across all trials, when presented with the Climate videos, ‘Climate’ audiences underperformed the ‘Philanthropy’ audience. In the most comparable trial (trial 4), they similarly underperformed both the ‘Philanthropy’ and ‘General’ audiences for this video. Aggregated across trials, the ‘Global Poverty’ audience appears to have done OK on with the Global Poverty related video. However, in the most comparable (4th) trial they did so poorly on this video that Facebook stopped administering it to them!
As reported in the Quarto, some messages show strong heterogeneity across audiences, while others were fairly consistent. The second best overall message, ‘100x impact’ did adequately on all audiences. ‘Learn’ showed some variation, doing pretty well on ‘Poverty’ and ‘Lookalike’ audiences, OK on ‘Animal’ audiences, but poorly on the ‘General’, ‘Philanthropy’, and ‘Climate’ audiences. The '6000+ people' and 'Bigger Difference' messages performed poorly on nearly all audiences, doing at best OK on a few audiences.
In the Quarto, we also graph HDI intervals (somewhat like confidence intervals) for the ‘results per unique impression’ by message. These intervals appear extremely narrow; the HDI’s do not overlap.[16] As each message is delivered at very nearly the same cost per impression (1.5-1.6 cents), the differences in ‘results per cost’ closely reflect the differences in ‘results per 1k impressions’.
Conclusions from video and age breakdown
Conclusions about Video Results by Audience
We are looking for collaborators with particular interests and expertise to help us design and analyze future campaigns as well as contribute to the ongoing analyses of trials we have completed (including the one discussed in this report). We are particularly interested in: adaptive trial designs to maximize ‘value of information’, Bayesian and robust simulation-based statistical analysis, meta-analysis and mixed models, social media advertising (e.g., Facebook ‘pixels’) and web analytics, and open data pipelines and visualizations (especially in R and Quarto). If you are interested in working with the EA Market Testing Team on this project or on similar research projects, please reach out to David Reinstein at [email protected].
In the future, we intend to test questions generated from these trials. For example:
Video content was manipulated across three ‘split tests’. (For a breakdown of treatment assignments by campaign, date, etc., see the tables in the dynamic document here.)
In all tests, the treatment assignment (which text and video were displayed to which of the users within the chosen audience) was determined by Facebook's algorithm. In the split test, the amount that each experimental condition was displayed[19] was set to equalize the total cost across conditions. The other features (e.g., ‘what cause video to show’ or ‘how to introduce each video’) were determined by Facebook's algorithm to optimize the rate of people leaving their emails (thus these were not balanced). None of the treatments were fully randomly assigned.[20]
The videos were adapted across the trials as we learned. First, we updated the factual video to be shorter for Trial 2, and then we tried videos of Luke holding up signs spelling out the voiceover in Trial 3 for all videos. In many of our analyses, we pooled data across trials, yielding greater statistical power.
Learn more about GWWC's efforts to identify strategies to engage people with EA by watching Grace's talk at EAGxOxford 2022, "What we're learning about spreading EA ideas."
These results are largely based on simple comparisons: we continue the careful statistical analysis of this trial in the Quarto dynamic document linked here, along with the analysis of other EAMT-linked trials.
Although there was some heterogeneity by audience, with this message doing poorly on ‘Poverty’, ‘Lookalikes’, and ‘Climate’ audiences in a comparable trial.
However, our confidence/credible intervals (reported in Quatro) are wider for this smaller group.
‘Dynamic documents’ combine code, text, and results, making it clear exactly how the results are produced.
We continue and extend the presentation and statistical analysis of this trial in the Quarto dynamic document linked here, along with other chapters analyzing our other EAMT-linked trials. (This is publicly hosted on the Github repo here, which contains all code and data for this project). We are eager to have you engage with that resource. You can leave comments on the Quarto with the embedded ‘Hypothesis’ tool. You can also engage with the Github, and reach out to us if you want to get further involved in this and other analysis and reporting.
Also see the resources on ‘implementing ads, messages, and designs’ that we are building in the public Gitbook here.
You can read more about how Facebook's ad auction works here.
See our walk-through on how to extract such data from Facebook here. (You can also import Facebook ad data into R directly via an API and helper tools; we intend to do this in our later work.)
We recognize (as noted above) these charts do not contain confidence/credible intervals, statistical tests, or metrics like ‘probability of superiority’. This presents particular challenges in this context; we have some preliminary analysis of this (as well as forest and ridge plots plots) in the Quarto, e.g., here and here.)
More information about treatment assignment by campaign is explained in the Appendix to this post.
We also set a ‘Retargeting’ audience, consisting of people who had already been on GWWC’s website. This was very small and obviously more promising than other audiences, so we do not include this audience in the analysis.
Facebook interests are determined by pages the user ‘likes’ and the content they engage with on Facebook as well as off the platform. Facebook's definition of ‘being interested’ is very broad and it might be different from what one would perceive as ‘being interested in something’ in the real world.
‘Lookalike’ audiences performed best overall, for both cause and non-cause videos. However, our confidence/credible intervals are wider for this smaller group. See Quarto for more details, e.g., the table ‘Results by audience; cause vs non-cause (and overall)’.
See the last table in the Quarto (go here and scroll to bottom); note that the ‘Climate’ audience did worse than the ‘Philanthropy’ audience when both saw climate videos, while the global poverty and philanthropy audience performed similarly. The table here focuses on the most comparable trials. If you filter by “video_theme = poverty” or “=climate” you see the results discussed below, suggesting each of these cause audiences underperformed the ‘Philanthropy’ audiences, even when presented their ‘own cause’s videos.’
Are these ‘real differences in the population?’ This is subject to other caveats about divergent delivery etc.
As discussed in our 'Audiences' section … aggregated across all trials, when presented the Climate videos, ‘Climate’ audiences underperformed the ‘Philanthropy’ audience. In the most comparable trial (trial 4), they similarly underperformed both the ‘Philanthropy’ and ‘General’ audiences for this video. Aggregated across trials, the ‘Global Poverty’ audience appears to have done OK on this video. However, in the most comparable (4th) trial they did so poorly on this video that Facebook stopped administering it to them!
The original videos in Tests 1 and 2 and the new videos in Test 3 used the same script; The videos in Tests 1 and 2 were animations. In Test 3, the videos had a voiceover instead and Luke held up signs with the words of the script.
I.e., factual vs. cause focus in test 1-3 and brand video vs. factual and cause based in test 4
See our comments about random assignment in the data collection and caveats section.
This is off the cuff, but I am sort of surprised all of this wasn't tested against CTAs specific to pledges and/or directly donating. Maybe you've done that previously and I've not seen that work or you haven't shared. 8 bucks a pop seems like a lot to me, for a cause promoter to spend, especially an EA cause, with no actual cause promotion. My (informed - I'm cause PR/marketing) guess is that if you associated your recruitment campaign with actual "donate" and "pledge" messaging with urgency around the cause, it would be just as effective as this in capturing emails - plus you get the whole actually being effective thing too. I'd also guess that if someone donates or pledges through your platform(s), they are far more likely to develop an actual relationship with GWWC than if shown a guide.
Was there a CTA (e.g., donate now) follow through with the ads linked to guides?
Last off the cuff: your 'results' should always end with a cause related goal for a cause when doing cause related marketing. Email capture is not a cause related goal. A captured email that leads to a pledge or donation - that's a goal and a result. Maybe you're tracking and measuring this on the backend. If not, I definitely would. A throw away email at 8 bucks a pop is not only expensive, but wasteful. You should find out if that captured email actually means something if you want to know if your campaign was successful.
PS - if mentioning "off the cuff" twice didn't communicate it, this is a draft opinion. I only read this post and once through at that. I could very well be missing things. I referenced email capture, but the same could basically be said for clicks as well.
Hi LB,
Appreciate your time in giving us some feedback!
For this years Giving Season campaign we're planning to test a donate CTA vs a newsletter sub CTA and track which causes more donations, pledges down the line. I agree that a donate CTA may fare better.
Also agree that perhaps a more cause focused campaigns could be effective, but this has to be balanced with the nature of GWWC as an org, our place in the effective giving ecosystem and what our vision is. This is something we are thinking about how best to do for our brand. I also think leaning into the community of GWWC and the vision we have is an interesting angle and am excited to see how this might compare to cause focused messaging.
The results we've shared here are just based on the advertising we did, but rest assured we are tracking donations and pledges off the back of this campaign - just not included in these results.
Hi GraceAdams,
Sure thing - please know its all meant constructively. I'm not issuing a complaint in anyway.
Hope something here was helpful! I've read this forum for ages without an account and only recently signed up to submit something and now I fear I am dangerously annoying. Ha! To be fair to me though, its rare that there is a cause comms post here. Good luck with your campaigning.
Thanks. Some solo responses; will give more detail once I touch base with the GWWC team.
We/they have done that in other contexts, see e.g., here.
But it’s a long funnel from a Facebook ad to a commitment to the very high-value ‘pledge’ outcomes (which are valued at tens of thousands of dollars by GWWC). And I believe that the email signup here is fairly meaningful, not something people will do lightly. It’s certainly more meaningful than simply clicking on a link.
Nonetheless, we are aiming to follow up in terms of understanding the value of these emails (are they ‘throwaways?’, do people stay in the system) and whether they lead to more substantial outcomes.
Compared to the value the EA community is putting on bringing new people into the fold, this seems like a bargain, but of course, it depends on the follow-through. (Not sure what you mean by ‘no actual cause promotion’ though; the ad, the page, and the giving guide are all promoting key EA-relevant themes.)
I'm sympathetic to your views, and I usually argue for incorporating costly (donation) outcomes in trials where possible. However believe the thinking was something like 'bring them into the funnel, don't put our hand out right away'.
Note that signing up for the giving guide involves going to the GWWC site (which has a range of CTA’s of the sort you are discussing). It also leads to email followups with further CTA’s like this.
And of course, the giving guide itself full of EA cause promotion, impassioned and reasoned arguments, and very impactful CTA’s, with links, e.g., below:
That’s a good point. We are tracking the ‘backend’ as noted above. We also hope to do more work in estimating and modeling the links between top and bottom funnel outcomes. This is something I’m very interested in (and pursuing related quantitative work involving value-of-information).
Thanks for your interest . And, as noted, i hope for more followup when the antipode wakes up.
Sure. Just a couple quick thoughts and clarification:
That is a long funnel - its an even longer funnel from those pledge landing pages to the actual effective causes. I think what I mean is to directly pitch the actual causes GWWC lists as effective (e.g., AMF). Like a FB video about Malaria that links directly to tailored AMF pledge landing page all about how this AMF pledge is super effective. Something like that. People donate from SM and search ads all of the time. And its probably also a lot easier to target specific causes through Meta's ad platforms than it is trying to profile people who might be down with preference sacrificing, no? Anyway, I don't think I was clear. I meant testing EA causes specifically and then growing that relationship from there. I think it would be an interesting outcome comparison to what you've just now reported on testing.
I am not sure its necessarily a valid cost effective measure to compare what you are doing to other EA org's audience acquisition costs. The United Way, which is much more analogous to GWWC, would probably be a better comparison. I highly doubt they have an 8 dollar email capture cost or clickthrough cost. New member cost, sure and then some, but that goes back to our now agreed to point, I think, about what 'results' actually are.
I mean the actual EA causes like AMF. GWWC is not a cause, neither is its pledge mechanism for that matter, really. Its a means to an ends for altruists. This and the dollar commitments to member acquisition EA groups commit to are probably a larger critique I should've written instead of my theoretical one that was likely straight up ignored (ha!) , but I think EA orgs could take a page from cause-related marketing history. For example, AMEX, when they paid to restore the statue of liberty in the 1980s, didn't actually do a ton of talking about AMEX. They cobranded the cause campaign, for sure, but it was all about the statue of liberty. And now everyone associates AMEX and the statue of liberty and not in a negative way and probably subconsciously for a lot of folks. And their numbers went boom during the campaign. The short point: the cause always comes first is the first rule of cause marketing. Altruists in particular would benefit from always putting the actual causes first, rather than the altruists club. If an altruist is doing too much talking about themselves (including their reasons and methods), people might start to doubt the whole altruist thing. Again, probably more meta than this very practical marketing campaign study, but I think its a pretty important perspective.
Possibly. Not sure 'preference sacrificing' is the right term though. Also, there is a big push within EA to bring people into the movement in conscious involved ways and not "just donate". GWWC is also trying to bring people in to consider direct work and other involvement.
This would indeed be an interesting comparison and possibly worth trying. But I'll wait to hear what/if the GWWC partners think about this. (Note that I am personally working on some trials involving things related to this, but not through GWWC.)
I don't see United Way as comparable to GWWC. Much less niche, much lower impact, substantially less of a commitment. But I don't know what their email capture cost is, and obviously not all email captures are alike. But do you have data on the United Way email or new member acquisition cost? This would be useful to have as a benchmark.
I think if you look at the content of
the video ads open the boxes and click the 'Video ads theme and content links here
or GWWC's home page and giving guide ...
... you will see a lot of powerful material about the actual causes.
This is also something we tested somewhat. We compared cause-themed videos to videos more generally discussing giving effectiveness. See here for some of the results across the most comparable trials. Of course, as you note, we are only reporting on a proximate metric -- people providing their emails/downloading guides. But at least for this outcome, the cause-specific videos did not consistently outperform the 'facts about giving' ones. (Although they did outperform these for some audiences, particularly animal-related videos for animal-interested audiences. On the other hand the animal videos performed particularly badly on the 'philanthropy-interested' audience.)
re: United Way - yes, it is a horribly ineffective organization compared to EA charities, certainly, but I was actually referencing their model - they are an intermediary between their selected causes and individual donors, like GWWC, they have a tailored list of causes to pledge to, like GWWC, and a giant part of their model is time period pledging, like GWWC. I just think their similarities would be a more apt comparison for cost effectiveness on email acquisition (or better, member/pledge acquisition).
I don't have their numbers, but its a pretty scrutinized organization and I think the comparison, frankly, is also campaign opportunity for an org like GWWC that is actually doing it better... maybe worth a poke around.