JoelMcGuire

1375Joined Jul 2020

Bio

I am a researcher at the Happier Lives Institute. In my work, I assess the cost-effectiveness of interventions in terms of subjective wellbeing.

Comments
82

But I'm interested in something more fine-grained than "total annual expenses, or even "program service expenses" (per tax returns). e.g.:
$A to train lay counsellors
$B / hour for facilitators * number of hours
$C operating costs for StrongMinds (SM)
$D for outreach to SM partners
$E for SM partner operating costs
etc

 

Unfortunately, I don't know if I can share any information beyond the pie chart I shared above. So I'll leave that for StrongMinds. 

Also, ~48% of clients were treated through partners in 2021, but does the methodology of working out cost effectiveness by dividing clients reached by SM expenses include expenses and operating costs of the partners?

We did our analysis before they shifted models, so we hadn't incorporated this. I don't think StrongMinds includes partner costs. This will be something we revisit when we update our StrongMinds CEA (expected in 2023). 

I see this as more of a concern for counterfactual impact. Where I see it as "StrongMinds got these organizations to do IPT-g, how much better is this than what they'd otherwise be doing?" But maybe I'm thinking about this wrong. 

Right, our concern is that if this bias exists, it is stronger for one intervention than another. E.g., say  psychotherapy is more susceptible than cash transfers. If the bias is balanced across both interventions, then again, not as much of an issue.  

I'm wondering if this bias your concerned with would be captured by a "placebo" arm of an RCT. Imagine a control group that receives an intervention that we think has no to little effect. If you expect any intervention to activate this "future hope" bias, then we could potentially estimate the extent of this bias with more trials including three arms: a placebo,  a receive nothing, and an intervention arm. 

Do you have any ideas on how to elicit this bias experimentally? Could we instruct interviewers to, for a subsample of the people in a trial, explicit something like "any future assistance [ will / will not] depend on the benefit this intervention provided." Anything verbal like this would be cheapest to test. 

I can maybe help with question 5, since the $170 figure originates from my analysis

I finalized the cost figures during COVID when their cost figures were very high ($400 per person). I tried to project what they'd be over the next 3 years (starting in 2020) and assumed it'd come down, but the costs have come down faster than I imagined. They now say they expect 2022 to cost 105 USD per person treated. 

They regularly update their cost and expense figures in their quarterly reports

 

And here's the general breakdown of their expenses according to their 2021 tax returns (page 10). 

PastedGraphic-1.png

A quick response -- I'll respond in more depth later, 

I love the idea of your RCT cash transfers vs. psychotherapy but I'm confused about a number of parts of the design and have a few questions if you will humour me .
 

To be clear, the planned RCT has nothing to do with HLI, so I would forward all questions to the authors! :)

And to be clear when you say "before and after measures", do you think this applies to RCTs where the measurement of an effect is comparing a control to a treatment group instead of before and after a group receives an intervention? 

Hi Jack, 

Right. The psychedelic work will probably be a more speculative and a lower-bound estimate. I expect we'll take the opportunity to cut our teeth on estimating the cost-effectiveness of research. 

Hi Rosie,

Thanks for your comment! Grief is a thorny issue, and we have different priors about it as a team. 

The reason I'm a little skeptical of this is first that it seems likely to me (disclaimer that I have not done a deep dive) that losing a child would increase the likelihood of depression and other mental illnesses, alongside other things like marriage disruption (e.g. Rogers et al. 2008, which highlights effects lasting to the 18 year mark). I don't think these effects will be accounted for by pulling out the estimate coming from Oswald & Powdthavee according to the Clark paper.

Could you clarify why you think the effects aren't accounted for? Is it because we aren't specifically looking at evidence that captures the long-term effects of grief for the loss of a child? 

If so I'm not sure that child-loss will have a different temporal dynamic than spouse-loss. I'd assume that spouse-loss, if anything, would persist longer. However, this was a shallow estimate that could be improved upon with more research -- but we haven't prioritized further research because we don't expect it to make much of a difference.  

Rather than underestimate grief, I'm inclined to think our grief estimate is relatively generous for several reasons.  

  1. We assume grief is equal for each household member. But do we expect a 2 year old child, if their sibling dies after childbirth, to be as sad as being depressed for 5 years? The effects on parents are probably larger. 
  2. We use evidence from high income countries. Will the effect on grief be large or smaller in low income countries? I'd guess that the effect of grief is less intense when death is less surprising and death is less surprising where it's more common which is the case in low income countries. To be clear, I'm not claiming how much less intense it is. 
  3. The team differs on this, but some of us think that if we account for the counterfactual value of grief, we'd decrease its effect. The reasoning from the pro-counterfactualists is that  everyone dies at some point, so presumably one will still be grieved for if they die at 50 instead of 5-years old. This is could get complex, because presumably the intensity of grief and number of people experiencing grief changes with the age of the person dying. 

All of this being said, I don't think it would change our priorities even if we had a sizeable change to the grief numbers, eg doubled or halved them, so we aren't sure how much effort it's worth to do more work.  

Hi David, 

You’re right that we should be concerned with the quality of published evidence. I discounted psychotherapy's effect by 17% for having a higher risk of effect inflation than cash transfers, see Appendix C of McGuire & Plant (2021). However, this was the first pass at a fundamental problem in science, and I recognize we could do better here. 

We’re planning on revisiting this analysis and improving our methods – but we’re currently prioritizing finding new interventions more than improving our analyses of old ones. Unfortunately, we currently don’t have the research capacity to do both well! 

 

Hi Nick, 

Thank you for sharing! This is valuable to hear. The issue of being primed to respond in a certain way has surprisingly not been explored widely in low-income countries.

We’re concerned about this, but to our knowledge, the existing evidence suggests this isn’t a large concern. The only study we’ve seen that explicitly tries to measure the impact of this was a module in Haushofer et al., (2020 section III.I) – where they find about a zero effect. If there’s more evidence we don’t know about, please share! 

Here's the excerpt (they call these effects "experimenter demand effects"). 

You also said "Also, we're comparing self-reports to other self-reports", which doesn't help the matter, because those who don't get help are likely to keep scoring the survey lowly because they feel like they didn't get help
 

Allow me to disagree -- I think this could help the matter when we're comparing between interventions. If this effect was substantial,  the big concern would be if this effect differs dramatically between interventions. E.g., the priming effect is larger for psychotherapy than for cash transfers. I.e., it's okay if the same bias is balanced across interventions. Of course, I'm not sure if it's plausible for this to be the case -- my prior is pretty uniform here. 

All that being said, I totally endorse more research on this topic!

We can't ignore how people feel, but we need to try and find objective ways of assessing it, especially in contexts like here in Uganda where NGOs have wrecked any chance of self reporting being very accurate.

I think this is probably a point where we disagree. A point we've expanded upon previously (see McGuire et al., 2022) is that we seriously doubt that we can measure feelings objectively. The only way we can know if that objective measure is capturing a feeling is by asking someone if it does so -- so why not just do that?  

 I am much more pro "find the flaws and fix them" then "abandon ship when it leaks" regarding measuring subjective wellbeing. 

 

Alternatively I'd be VERY interested in a head to head Cash transfer vs Strongminds RCT - should be pretty straightforward , even potentially using your same subjective before and after scores. Surely this would answer some important questions.
 

This may be in the works

This is implicitly 0.42 WELLBYs per $1k if we go with the HLI adjusted figures, or it's 1.67 WELLBYs per $1k if you take GiveWell's income figures at face value. 

Again, GiveWell doesn't explicitly model the morbidity effects other than by inflating the value of malaria prevention's life-saving and income-increasing effect by 9%. We didn't tinker with the supplemental charity-level adjustments, supplemental intervention-level adjustments or their leverage/funging adjustments because that is, I expect, a whole can of worms. Because we kept these adjustments that GiveWell uses to tweak the value of income and life --  the morbidity effects that GiveWell implicitly incorporates, we implicitly include it as well. 

Basically, if you think that the morbidity effects should merit a different adjustment than 9%, we don't account for that. If you're satisfied with 9%, then it's already accounted for, just in a weird opaque way as part of GiveWell's suite of subjective adjustments. 

What worries me here is that you don't need to be a totalist to have these concerns.

Right, I should have clarified that the gnarly thing with totalism is considering the effect on all future 14k+ generations and the likelihood they exist, not just the higher-order effects on the presently existing population. 

However, I'm not the philosopher, so Michael may disagree with my sketch of the situation. 

Load More