All of jonathonsmith's Comments + Replies

More Thoughts (and Analysis) on the Mercy For Animals Online Ads Study

Vegan Outreach ran its first annual Leafletting Effectiveness Survey (LES) last fall and we had a dismal response rate as well (around 2%). We were offering $5 incentives for people to take a 2-part survey, where Part 1 was filled out immediately and then an email was sent out two months later to complete Part 2 and claim their gift card. We've been running small response rate studies since then to figure out what kind of incentives we need to hit our targets, but we're seeing significant variation based on what city / state we're operating in. This is mak... (read more)

1Peter Wildeford5yI have not. I don't believe we collected geographic data (it's not in the public data set provided), but you could check with Krystal at MFA. - It’s hard to say for sure, but I suspect it was because the pilot study was not run for very long, so we inadvertently selected for more enthusiastic people.
More Thoughts (and Analysis) on the Mercy For Animals Online Ads Study

As I've said elsewhere, I'm skeptical that the approach to take is to do more such RCTs. I worry about us having to spend extremely large sums of money for such things.

It's probably a good idea to consider the global amount of money being spent on an AR intervention when evaluating the cost to investigate it. Like how much money is being spent across the different AR orgs on FB ads? If a proper study costs $200K and there is only $500K a year being spent globally, then it's hard to see the value proposition. If the total being spent annually is $20M, th... (read more)

3Joey5yI have heard that farm animal welfare as a whole is in the $10m-$100m range, so I would be surprised if something like online ads was $20m a year. That being said, it's worth accounting for long term effects. For example, if online ads were proven not to work for $100k and only $200k gets spent on it a year, the first year might seem like a waste, but if over the next ten years 50% of funding for online ads moves to more effective interventions, this definitely makes it worth it. Additionally, if something is proven to work, then the amount of total AR funding that goes to it could increase to well past the amount it's getting now. For example, if online ads get strong evidence showing they work, they might get $500k a year instead of $200k and other less proven interventions might get less.
MFA Ad Study Targeting Former Vegetarians

Lot of great points here and angles to follow up on.

I do want to re-assert that enormous 80% recidivism rate, and how strong these (very valid) counter-narratives would have to be to defeat our original assumptions. Consider:

  • A huge number of people would have to actively curate their Facebook profiles to give current-vegetarians the majority in these groups.

  • A curiously large number of current-vegetarians would have to click on an ad that claims to "help you eat vegetarian again" and order a "Vegetarian Starter Guide" to make the ma

... (read more)
1Owen_Cotton-Barratt5yI'm not sure your interpretation of the 80% number is correct. As you originally presented it, 80% of people becoming vegetarian stop within a few years. However: * Sampling the population at a given time (e.g. the time the facebook group has been running) will be more likely to catch long-term vegetarians, because they make up a much greater proportion of vegetarian-years. * Facebook hasn't been going that long and I don't think has reached equilibrium. You may have collected a reasonable number of people in the couple of years after they joined the group, who would eventually be recidivists but aren't yet (this could actually be a pretty good group to target). * Possible selection effects will mean even at a given time long-term vegetarians are more likely to join a facebook group, if it's more of a core part of their identity (this could also go the other way).
MFA Ad Study Targeting Former Vegetarians

I think I follow you here. Facebook's ad engine is more likely to target people that have liked several of our selected groups / terms. Liking multiple groups indicates stronger identification with being veg, so these people are more likely to still be veg, as opposed to people who liked one but not the others. Is that right?

MFA Ad Study Targeting Former Vegetarians

Excellent thoughts here. As I mentioned in another comment, a follow up study could probably handle that second issue by including a question asking if the requestors of the VSG are current, former or (aspiring) new vegetarians.

This would probably shed some light on your first point as well. If most of the people requesting the VSG identified as current veg, then that would indicate either the ads aren't working at enticing former vegs to try again, or there just aren't any former vegs in the audience. Either of these would be enough to kill this as a strategy for reengaging recidivist. Although, this would open up the question of why so many current vegetarians are interested in a Vegetarian Starter Guide?

0zdgroff5yI think it's a low enough rate that it's certainly plausible it's disproportionately current veg*ns. If you see the comments above about identity and why recidivists may be more likely to revert again, I would caution against making too large decisions based on this.
MFA Ad Study Targeting Former Vegetarians

It's definitely possible that some current vegetarians might have requested the the Vegetarian Starter Guide. A follow up study could probably parse out this variable by having a simple required question for obtaining the VSG (along with their email) asking if they are currently veg, former veg and interested in trying again, or never been vegetarian at all.

1zdgroff5yOf course, we know these sorts of things are not always reliable, but I agree that's a good idea.
MFA Ad Study Targeting Former Vegetarians

The only reason I wouldn't put that document out publicly is because it wasn't written for wide release, so maybe Nick would want to clean it up before having it shared around. I know I usually spend more time polishing the look and language of a document that I intend to be passed around publicly. But that is the only reason, we're definitely happy to share any details people are interested in.

1davidc5yImage is working for me now.
MFA Ad Study Targeting Former Vegetarians

Nick wrote up a pre-study plan that I can send your way if you (or anyone else) would like to see it. Really though, it was a pretty simple study. We targeted people who liked one or more of the following terms / pages (below) with ads encouraging them to give eating veg another shot. But definitely let me know if you have any specific questions and Alan or I can get you the details. As an aside, can you confirm for me that the images are showing up now?

Terms used to target study audience:

  • Vegetarianism
  • Vegetarian Cuisine
  • Lacto Vegetarianism
  • Ovo-lacto Vegetarianism
  • Semi-vegetarianism
  • Flexitarianism
  • Vegetarian Times
  • VegNews
3davidc5yImage is not showing up for me still. Is there any reason to share those details privately instead of being transparent in public? Thanks for letting us know about this study!
MFA Ad Study Targeting Former Vegetarians

Hey guys, sorry I didn't realize this had posted. I was still editing the draft and just now finished cleaning it up only to find it went live last night (damn!). Here are some of the numbers (extracted from an email Alan sent me summarizing the results):

"To get one person in the US to pledge to go vegetarian and order a VSG it usually takes $5-$8 but in this trial we were able to get VSG orders for an average of only $2.65. This means that 2-3x more people ordered a VSG than if we were to target the general US population with the same amount of adver... (read more)

Effective Altruism Outreach winter fundraiser

A quick note on what encouraged me to donate to EAO.

I navigate robotic spacecraft to destinations in deep space at JPL. If you're trying to get somewhere like Jupiter or Saturn, the most important course corrections you can make are right after launch. We always have a crack team of analysts closely monitoring a spacecraft just after it leaves Earth, because the energy required to change the spacecraft's heading grows exponentially with time; point in the wrong direction too long and the mission is lost.

EA is moving really, really fast, and small adjustm... (read more)

1Kerry_Vaughan6yThanks so much for your support Jonathon!
Don't sweat diet?

I find these discussions of moral offsets somewhat disturbing, re: Tom_Davidson's third point. Can we host a dog fighting ring at EA Global next year as long as half the buy-in goes to the Humane League? Can we get trafficked children to cook our food as long as we give a nice plump sum to SCI?

I think the analysis is fine, and it's good to know the real impact of certain actions (like going vegan). But then to take it a step further and say, well, I can just skip acting morally in this case and offset that with a donation seems to miss the mark. How far are we willing to go, as a community, down this road, and where do we draw the line?

1kbog6yThe only clear and decisive way out is to accept that your life should be maximally ethical. So you shouldn't go around pursuing wanton acts of vice, but you should make whatever decisions are necessary to maximize your overall ethical productivity.
The term "Vegan" needs to evolve

That's definitely an option, but getting new terms to stick is really tough. Pragmatically, I think it makes more sense to push Vegan in a more useful direction then try to organize around a brand new term. Plus, there are a lot of activists out there with really great Vegan tattoos, it'd be a shame to drop it from the lexicon.

A Defense of Normality

Great article Eric. I had similar thoughts at EA Global this year. The common tagline for EA that I heard repeated was "1.) Find out how to do the most good, 2.) Do it", and while I agree with this sentiment, I would add a couple un-sexy modifiers; "1.) Find out how to do the most good with the time and money you have decided to donate, 2.) Do it."

I really appreciate and look up to the people who are all-in and looking to maximize the good they can do with their lives. One of the reasons I like to go to EA events is to have my own altr... (read more)

TLYCS Pamphlet Pilot Results

They must have some visibility into how many people are donating via their website, because they release yearly estimates for money moved. I'm not sure exactly how they go about doing this; it can't be via TLYCS website analytics though, so maybe they work with the charities themselves to track donations originating from TYLCS. I only partnered up with them for this study, so unfortunately I don't have any details.

0Jon_Behar6yTLYCS's (very) rough estimate is that on average each "donate click" equates to ~$100 in donations. However, that's on overall average for our site, so my guess for a new visitor, currently enrolled in college, would be significantly lower. As a reminder, TLYCS also expects to generate a multiplier on its expenditures, so the true breakeven cost would have to be <33% or so of the expected donations. Sorry to take so long to respond to this!
TLYCS Pamphlet Pilot Results

Yeah, one of the main "unmeasurables" of the pilot was how much more of an impression a pamphlet, handed out by a real volunteer, can have on someone over an online add. This goes into the "touchpoint" theory, where it may take multiple exposures to an idea before someone bites; and maybe getting a pamphlet is a much stronger "touchpoint" than seeing an online add. I personally think that getting handed a pamphlet by a non-paid volunteer is a really powerful thing; and that's the reason that (again personally) I think that it ... (read more)

TLYCS Pamphlet Pilot Results

Unfortunately, we could only track website visitors as far as clicking a "donate" button on TLYCS "Where To Donate" page ( After clicking the button they are directed to the individual charity website, so we don't have visibility into if they actually followed through with making a donation, or how much that donation was.

That being said, we only had one visitor that plausibly came from the pamphlets that clicked a donate button; if we had paid closer to Vegan Outreach prices for the pamp... (read more)

1jayd6yDoes LYCS have any estimates which speak to that, even tangentially, for instance for it's general impact evaluations?
TLYCS Pamphleting Pilot Plan

Awesome, thanks for diving to this level of detail. You mention a lot of good points, some of which we've thought of, some not. I've started emailing statistical consulting companies, we'll see what comes back.

I do want to pose this question in another way that I think reflects more accurately my doubts about the necessity for a statistician. I mean, I definitely agree having someone on board with that skill set would be nice ... so would having a world class add agency designing the pamphlet, and a small army of volunteers to hand them out, etc. But is it... (read more)

0Bernadette_Young7y(Sorry for taking so long to reply) A statistically sound study design is important for two major reasons I can see. Firstly it will maximise your chance of answering the question you are trying to answer (ie be adequately powered, have robust confidence intervals etc). But in addition it will help make sure you are studying what you think you are studying. Giving adequate consideration to sampling, randomisation, controls etc are all key, as is using the correct tests to measure your results, and these are all things a good stats person will help with. Having a 'precise' result is no good if you didn't study what you thought you were studying, and a small p value is meaningless if you didn't make the right comparison. Regarding why I think bad data is worse than no data, I think it comes to a question of human psychology. We love numbers and measurement. It's very hard for us to unhear a result even when we find out later it was exaggerated or incorrect. (For example the MMR vaccine and Wakefield's discredited paper). Nick Bostrum refers to 'data fumes' - unreliable bits of information that permeate out ideas and to which we give excessive attention.
TLYCS Pamphleting Pilot Plan

Actually it does appear you can hire a statistician like a lawyer or accountant, I'll be damned lol. I just typed "statistical consultant" into Google and got like a million hits. I would love a personal recommendation if you have one though.

TLYCS Pamphleting Pilot Plan

Thanks Bernadette! Other people have suggested consulting a statistician, but it's not been clear to me precisely what she is supposed to do. I went through some lengths to be as specific as possible in our plan about what our data is, what we expect to see, and how we plan to calculate our success criteria (e.g. creating dummy numbers and producing working code that runs through our calculations). Can you maybe poke some holes in our approach so that I get a better idea of what a statistician would be bringing to the table?

Also, do you know how one goes a... (read more)

4Bernadette_Young7yYou're right there are definitely statisticians for hire. All my experience is in health research, and our stats people are pretty oversubscribed I'm afraid. If you or anyone doing this has an academic association, then I would pursue your institutions for possibilities. The first hole I would poke is that your control is not defined. On page 5 you say you can 'vary the strategy' for defining the baseline, and while I understand your reason for this is to avoid using an unrepresentative baseline if something weird is happening, varying the control = data mining. From the outset you are giving yourself multiple (non-independent) tests to do to find an effect. I would suggest you define your baseline robustly as the corresponding days of the week in a lead in period. I suggest you run that analysis now, and if you find the data varies wildly with how long a lead in period you choose, then that is something to discuss with TLYCS (are there factors that might be confounding, are there time periods to exclude) and with your statistician. (Also, regarding baseline, forgive me if I missed it, but can you use ISPs to limit this to visits from California? that would seem to be a good way to exclude a lot of noise. Also increases in non-Californian visits after distribution might help signal about other confounded (like a media story)) The second is that you need to have a value (even estimated) for your measures at baseline and you need a power calculation. If you find a null result, is that a mission killer? If you're not powered to find effects you would care about, then you need to consider a bigger pilot/further study. You make an oblique reference to power in that you state you don't think the number of donors/pledgers will be enough to be detectable, hence a composite measure for donors. This is where I think you get a it vague - the conversion ratios you choose will have a massive effect on that measure, and the plausibile values differ by orders of magnitude.
2jonathonsmith7yActually it does appear you can hire a statistician like a lawyer or accountant, I'll be damned lol. I just typed "statistical consultant" into Google and got like a million hits. I would love a personal recommendation if you have one though.
TLYCS Pamphleting Pilot Plan

Thanks Ben. Yes, I know that you can quantitatively define this, and it's something we may look into more in the future. We decided against pursuing it right now because 1.) none of us know how to work this problem and we would have to sink some time into learning, 2.) we're pretty sure we know the gist of the answer (my previous comment), and 3.) we're not really in a position to change our strategy based on the results anyway. I'm hoping to be able to publish our actual data after we run the pilot, so if there are any enterprising statistical EA's out there that want to sink their teeth into it, we'd be delighted.

4Bernadette_Young7yWell done in the work so far. I have some experience in study design (including formal postgraduate qualifications in study design), and I think it's vital that you consult someone with statistical experience before you proceed. I just can't express it strongly enough: it's absolutely critical. I understand you are concerned about resources, but you are prepared to print 6500 leaflets and take days of time to distribute them in the hope of generating data, which you are going to analyse.You shouldn't proceed unless you're also prepared to do this. It's not simply a matter of whether you need to tweak the intervention. You need to make sure that your control is properly defined (it's currently very vague, and susceptible to data mining). You also need a clear statement about what kind of effect you are looking for, what tests you'll apply to find them, and how big the effect would need to be to be detectable by this study. It's really laudable to be seeking good evidence to guide recruitment interventions, but it's counterproductive to produce bad data, and it undermines our evidence based message if we can't get the essentials right. This stuff is really hard.
TLYCS Pamphleting Pilot Plan

Very good point here. As I mentioned in another comment, I think we will have strong statistics for our baseline numbers, because we will be mining a year-plus of Google analytics data to generate them. So we should be able to tell if the individual distributions deviate significantly from the baseline. The way we have things planned now, we will be handing out a large number of pamphlets on a small number of days. In a best case scenario, we will get really large deviations from the baseline, so even if we're not able to hone-in on the true mean and stand... (read more)

2Ben_Kuhn7ySorry for not making this clear in my original post, but you can actually explicitly calculate your power to detect an effect of a given size once you know your study design and the tests you'll be performing (see e.g. here [] for a basic one although I'm not sure whether you'll be able to use it; it depends on your model). This is much better than trying to eyeball whether you have enough power unless you have a very good intuition for statistical significance.
TLYCS Pamphleting Pilot Plan

We have a custom URL to a quiz page, but as I mentioned in another comment, it's uncertain right now how many interested people will actually type in the full URL address rather than just Googling "life you can save". I think that the hits to the custom page will be a good indication of "strong engagement", as in, people were really into what they read enough to type in a URL.

0Peter Wildeford7yThat makes sense. To do this successfully, you'd probably have to make a new brand with a new domain, and see how much traction that new brand gets. Probably not worth it...
TLYCS Pamphleting Pilot Plan

I think we'll be able to get a standard deviation, as well as mean, for the baseline values we compare against, which should be helpful to determine if the individual distribution results are significantly different than the baseline rates. I don't think we'll have enough distribution days in the pilot to be able to get the same for the pilot numbers (e.g. we won't be able to tell if the results of individual distributions is typical of all distributions), but that seems like something we could accumulate over time if we proceed with the program.

There is ... (read more)

TLYCS Pamphleting Pilot Program

This is a really good point. Yeah, the scaling model is to have local TLYCS chapters organizing volunteers to do this as a regular, rolling semester activity. I hadn't really considered myself a confounding variable in this sense, because I'm definitely not a master pamphleteer. I'm an engineer by trade, and if this program takes off, I'll eventually just be another volunteer in the LA area that helps hand out leaflets occasionally. We're also thinking about splitting crews on Friday distribution days - so I would have a crew that hits up two universities, and there would be another volunteer crew hitting up two different campuses. Any thoughts on this?

TLYCS Pamphleting Pilot Program

That is definitely the intention. We are really hoping that the data we gather will be useful to other orgs considering a similar program, which was part of the motivation for posting up here ahead of time to get feedback.

TLYCS Pamphleting Pilot Program

Pretty sharp! If I had seen this before, I definitely would have passed it along to our designer as something to work from.

TLYCS Pamphleting Pilot Program

That would be great! I'll connect with you on Facebook and we can open up a line of communication there.

TLYCS Pamphleting Pilot Program

Yeah, if you give me the contact info of a statistician that you recommend that would be great. I don't know if we have the budget for it, but I would definitely reach out.

5Ben_Kuhn7yI'm checking for people who would be interested in doing it pro bono. If that doesn't work, I'm 99% sure you can find some people to fund a couple consultant-hours. Not to put too fine a point on it, but if the alternative is TLYCS designing the experiment themselves, this is pretty much like running a charity that spends nothing on overhead. It looks good on paper, but in reality, that last bit of money is a huge effectiveness multiplier.
2Peter Wildeford7yI'd consider funding this if it's "worth it" and not too much money. I'm sure others would as well.
TLYCS Pamphleting Pilot Program

Really good point here; I was a fan myself of the online Giving Game, but that would be hard to scale with the program without securing a donor willing to finance it at a pretty large level.

0Peter Wildeford7yIf the hook is worth it, how expensive would it be to scale and hard would that be to finance? I suppose if the initial pamphlet run is worth it, you could then A-B test it with a Giving Games pamphlet.
TLYCS Pamphleting Pilot Program

Some really good points here. I never considered that handing out the leaflets only on Fridays might skew the results (I just happen to have every other Friday off, thanks California), I'll have to think that through. And it would definitely be a good idea to have a "Where did you hear about the pledge?" question on the pledge site, I'll check into that as well.

I'm not sure what our initial run on the pamphlets will be, but I'm thinking in the 5K-15K range. I haven't done any analysis to figure out how many we'd need to hand out to get good stati... (read more)

6Ben_Kuhn7yPlease talk to a real statistician if you're designing an experiment! Random Internet people picking your design apart is actually pretty good as far as review goes (if they're the right Internet people), but actual statisticians are orders of magnitude better. Experiment design is very tricky and good statisticians are aware of both lots of tools to make your job easier, and lots of pitfalls for you to avoid. To quote Ronald Fisher:
TLYCS Pamphleting Pilot Program

Just took a look at their website, very cool stuff. You suggesting I email them and get their feedback on our plan?

0Giles7yDefinitely. Some of the team at least are EA insiders and lurking on this very forum, and they'll already know about TLYCS for sure.