I'm Sean Mayberry, and I’m the Founder/ Chief Executive Officer of StrongMinds. I will spend time on the Monday after the Thanksgiving holiday answering questions here (though I may get to some questions sooner).
A little background information about me:
- I founded StrongMinds in 2013. We are a social enterprise/NGO that treats depression in low-income women and adolescents by providing group interpersonal therapy (IPT-G) delivered by lay community health workers. StrongMinds is the only organization scaling a cost-effective solution to the depression epidemic in Africa.
- Our model developed from the findings of a randomized controlled trial in Uganda in 2002 that had remarkable success in treating depression with group interpersonal psychotherapy (IPT-G). The study, by researchers from Johns Hopkins University (JHU), used lay community workers with only a high school education.
- I left my position as the CEO of a global antipoverty organization and founded
StrongMinds, concentrating in Uganda, the site of the previous randomized controlled trial. I used my family’s savings to accomplish this and volunteered full-time for the first 18 months until supporters were identified. We would seek out individuals with an interest in being data-driven, entrepreneurial, people-focused, passionate, open, and collaborative. Those traits eventually informed the core values of the company culture at StrongMinds. - StrongMinds has now treated over 160,000 women with depression to date in Uganda and Zambia. On average, 80% of the women we treat remain depression-free six months after the conclusion of therapy. When our clients become depression-free, they can work more, and their kids eat and attend school more regularly. They also report that they no longer feel isolated and have people to turn to for social support. By the end of 2022, we will have treated over 210,000 women and adolescents through our work.
- Drawing on evidence from over 80 academic studies, Happier Lives Institute has found that the group interpersonal therapy provided by StrongMinds is almost ten times more cost-effective than giving cash to people in extreme poverty (a standard benchmark for aid effectiveness).
- I have been honored to present at a few Effective Altruism events. We love that the
the community has taken such an interest in StrongMinds’ approach centered around data collection, transparency, cultural competence/appropriateness, and human well-being.
Please ask me anything! I look forward to answering all of your questions.
UPDATE: I'm sorry for the delay in responding, but I wanted to take the time to be thoughtful and thorough with each of my responses.
Many thanks for doing this AMA!
I'm personally excited about more work in the EA space on topics around mental health and subjective well-being, and was initially excited to see StrongMinds (SM) come so strongly recommended. I do have a few Qs about the incredible success the pilots have shown so far:[1]
[Edit: I note that the 99% figure in the phase 2 trial was disregarded, but the 94% figure in phase 1 trial wasn't, despite presumably the same methodology? Also curious about the separate analysis that came to 92%, which states: "Since this impact figure was collected at a regular IPT group meeting, as had been done bi-weekly throughout the 12- week intervention, it is unlikely that any bias influenced the figure." I don't quite understand how collection at a regular IPT group meeting makes bias unlikely - could you clarify this? Presumably participants knew in advance how many weeks the intervention would be?]
Thanks again!
(Commenting in personal capacity etc)
[Edited after Joel's response to include Q7, Q8, and an update to Q1c and Q5, mainly to put all the unresolved Qs in one place for Sean and other readers' convenience.]
[Edited to add this disclaimer.]
[Edited to include a link to a newer post StrongMinds should not be a top-rated charity (yet), which includes additional discussion.]
Apologies in advance if I've missed anything - I've only briefly skimmed your website's publications, and I haven't engaged with this literature for quite a while now!
Quick primer on NNT for other readers. Lower = better, where NNT = 1 means your treatment gets the desired effect 100% of the time.
SM's results of 95% depression-free (85% after the 10% adjustment for social desirability bias) give an EER of 0.15 after adjustment. By a more conservative estimate, based on this quote (pg 3): "A separate control group, which consisted of depressed women who received no treatment, experienced a reduction of depressive symptoms in only 11% of members over the same 12-week intervention period" and assuming all of those are clinically significant reductions in depressive symptoms, the CER is 0.89, which gives an NNT of 1 / (0.89 - 0.15) = 1.35. The EER can be adjusted upwards because not all who started in the treatment group were depressed, but this is only 2% and 6% for phase 1 and 2 respectively - so in any case the NNT is unlikely to go much higher than 1.5 even by the most conservative estimate.
They also concluded: "We did not find convincing evidence supporting or refuting the effect of interpersonal psychotherapy or psychodynamic therapy compared with ‘treatment as usual’ for patients with major depressive disorder. The potential beneficial effect seems small and effects on major outcomes are unknown. Randomized trials with low risk of systematic errors and low risk of random errors are needed."
See Appendix B, pg 30. for more context about what the PHQ-9 scoring is like.
As pointed out in the report (pg 9):
% clients reached by partners:
20392 / 42482 in 2021
33148 / (33148+8823) in 2022
Thanks for these questions! I appreciate the time you took to really look at our data, and I think some of the questions will help us ponder what we need to be looking at next within StrongMinds. Please note, as the Founder, I'm not a researcher or clinician by trade, so my answers may not be as granular as you would hope, but I'll do my best to respond. I'm going to tackle the first four tonight.
No worries! I should say that I've spent less than 3 hours looking through SM/HLI documents around this, so I'm not highly confident about most of these points. I have a lot of respect for anyone who is trying their best to make as much impact in the world as they can - thank you for all your work thus far, and thank you for engaging with all the questions!
I should also clarify that the digging was largely prompted by HLI's strong endorsements:
And while this was a result of over 3 years and 10,000 hours of work, I generally aim to be more hesitant to take such strong claims at face value.
But I mention this because I want to emphasise that even if it's the case that after this conversation, I decide that I'm not personally quite convinced that StrongMinds is the single most cost effective way to help other people, it doesn't mean this is a reflection of the effort you have and continue to put into SM! It doesn't necessarily mean SM isn't a great charity. It doesn't mean it's not possible for StrongMinds to be the best charity in the future, or be the best under different philosophical assumptions. It's just really hard to be the most cost effective charity.
And I'm mindful that this conversation has been possible precisely because of your shared commitment to transparency and willingness to engage, which I have a lot of respect for. We are both on the same team of wanting to do as much good as we can, and I hope you interpret this barrage (apologies!) of questions in that light.
Lastly, I'm also happy to continue via email and update folks later with a summary, if you think that would be helpful for getting answers that you may not be able to answer immediately etc.
With that in mind, some followups:
1) Just re-flagging the question RE: bias, though as you pointed out, this may be better suited for a researcher on the team / someone who was more in-the-weeds with the research:
a) What's the justification behind the 94% figure not being found to be invalid when the 99% was? Was it based on different methodology between the two pilots, or something else? As far as I can tell, the difference in methodology RE: recording PHQ-9 scores was that in phase 1 these were scored weekly from week 5-16, with a post-assessment scoring at week 17, and for phase 2, these were done biweekly from week 2-12, with a post-assessment at week 14. It's not clear that this difference leads to bias in one but not the other.
b) Also curious about the separate analysis that came to 92%, which states: "Since this impact figure was collected at a regular IPT group meeting, as had been done bi-weekly throughout the 12- week intervention, it is unlikely that any bias influenced the figure." I don't quite understand how collection at a regular IPT group meeting makes bias unlikely - could you clarify this? Presumably participants knew in advance how many weeks the intervention would be?
2) I took the 10% from StrongMinds' 2017 report (pg 2), not an HLI analysis (though if HLI independently came to that conclusion or have reviewed it and agreed I'd be interested too):
I couldn't find a justification of this figure in that report or any of the preceding reports. (admittedly I just very quickly searched for various combinations of 10/85/95% and didn't read the entire report)
3) Makes sense - looking forward to the results of the RCT! I assume it will be independent and pre-registered?
4) Thanks! Just looked in a bit more detail - in Appendix A (pg 30) it says:
"Use of lay community workers as the IPT-G facilitators, or Mental Health Facilitators(MHFs)
In Appendix E (pg 33) it says:
a) Just confirming that "lay counsellor" is the same as "IPT-G facilitator" and "mental health facilitator"? If not, what are the differences? How much training do they get and what's their role in the intervention etc.
b) How does StrongMinds select for empathy? E.g. questionnaire, interview, etc.
c) What does it mean to be a "certified IPT-G expert"? For example, it sounds like there are a lot of various levels of certification. From a quick google the best match I found for the description of the training was this, which suggests a "certified IPT-G expert" is someone who has completed this specific 6-day course, i.e., with a level A certification? (Happy to be corrected - just couldn't find any details of this). If true, am I understanding correctly that the StrongMinds lay counsellors / mental health facilitators take a 10 day training course which is delivered by someone who has taken a 6-day course? Do the certified IPT-G experts play a role in SM other than the initial training of lay counsellors?
d) What does it mean to be a "mental health supervisor"? What's their role in SM?
e) [Minor] I just realised the appendix said MHFs require at least a high school diploma, contra what you said earlier: "in fact, they don't even need to have a high school degree". I assume this was just a more recent change in policy. Not a big deal, just clarifying.
5) I had another question, which came up as I was going through the tax returns Joel linked to work out the cost per client reached - in the tax return it says
This didn't exist in the tax returns before 2019, but came up every year from 2019 onwards.
a) Was there a change in model in terms of revenue streams or business model for StrongMinds - if so, what changed?
b) You'll probably cover this in some of the other questions, but how do the partnerships work? Do partners pay you for the year of training? What does this training look like?
c) Are there other revenue streams that StrongMinds have outside of donors / grants? (To be clear - I don't have an issue with StrongMinds being a social enterprise, just wanting to clarify so I have the facts right!)
(commenting in personal capacity etc)
Thank you! I appreciate your curiosity, and I'm not put off by the questions or anything; it's just many of them are not in my area of expertise, and this happens to be a pretty busy time of year at StrongMinds. It may take some time to fully gather what you're asking for. We aren't a large research institute by any means, so our clinical team is relatively small. Additionally, some of the work you are referencing is nearly a decade old, so we have shifted some of the ways we operate to be more effective or better based on our learnings. That said, I will dig back in when I can to help answer your additional questions via email or direct message.
To answer the remaining four from your original note to close the loop:
5) Since HLI generated the $170 figure, they have the best information on that particular breakdown, but I am collecting the most recent info on our CPP for another question, and I will share that with you later this week when I have the updated numbers.
6) As mentioned above, we are currently in the process of assessing the right questions and framework for an RCT looking at the results and impact of our therapy model. We are hoping to be able to launch the RCT late in 2023.
7) We switched our model to teletherapy to continue to treat clients during the pandemic lockdowns. It was not ideal, but we wanted to continue reaching as many women as possible despite the challenges. Though it proved tricky in some cases to reach our target demographic, we did find that some women preferred the flexibility teletherapy offered them. For the most part, we have switched back to our original model, but we still see some groups via teletherapy in Uganda. All research shared publicly from our initial year using teletherapy can be found here.
8) We track individuals that attend most of their therapy sessions, as we saw that the effects of therapy were still statistically significant and that attending additional sessions did not produce incremental impact. Due to the individual roles and responsibilities of the women that attend, it's sometimes challenging for them to make it to all 12 sessions.
Thanks again for the questions!
I can maybe help with question 5, since the $170 figure originates from my analysis.
I finalized the cost figures during COVID when their cost figures were very high ($400 per person). I tried to project what they'd be over the next 3 years (starting in 2020) and assumed it'd come down, but the costs have come down faster than I imagined. They now say they expect 2022 to cost 105 USD per person treated.
They regularly update their cost and expense figures in their quarterly reports.
And here's the general breakdown of their expenses according to their 2021 tax returns (page 10).
Thanks for this Joel!
RE: Q5 - sorry, just to clarify, I was interested in a breakdown of the $170 figure (or the 109 / 134/ 79 figure in the cost-per-patient graph). What does it consist of?
On skimming the HLI report it says: [1]
But I'm interested in something more fine-grained than "total annual expenses, or even "program service expenses" (per tax returns). e.g.:
$A to train lay counsellors
$B / hour for facilitators * number of hours
$C operating costs for StrongMinds (SM)
$D for outreach to SM partners
$E for SM partner operating costs
etc
I'm mindful this is asking a lot of info, sorry! I just assumed it'd be readily available, but it looks like you've just deferred to SM here.
I had a very brief look through the tax returns - per the tax returns you linked, the total expenses for 2021 come to 5,186,778. Per the quarterly report you linked, the total clients reached in 2021 was 42482. This means the $ per client figure should be $122? But that's not the $134 figure reported, so I'm probably doing something wrong here.
Also, ~48% of clients were treated through partners in 2021, but does the methodology of working out cost effectiveness by dividing clients reached by SM expenses include expenses and operating costs of the partners? Q2 results (pg 2) suggest StrongMinds is on track for ~79% of clients treated through partners in 2022. If the expenses of the partners aren't covered by SM but the clients reached are then this will make SM look more cost-effective than it is.
I also saw in the HLI report that SM defines treated patients treated here as "attending more than six sessions (out of 12) for face-to-face modes" - is this also the definition for the treatment? i.e. how did the pilot assess the effectiveness of SM for people who attended 7 sessions and then dropped out?
Do you know the answers to the other Qs too? If so, I'd be interested in your take as well! But also no worries if you prefer to leave it to Sean (I've edited the comment above to incorporate these Qs).
Sorry if I missed it, I just ctrl+F'ed 170 in the forum post you linked which didn't give me a result, so I skimmed section 5 in the full HLI report. I also looked at the Q report and the tax returns but it doesn't quite answer the question.
Unfortunately, I don't know if I can share any information beyond the pie chart I shared above. So I'll leave that for StrongMinds.
We did our analysis before they shifted models, so we hadn't incorporated this. I don't think StrongMinds includes partner costs. This will be something we revisit when we update our StrongMinds CEA (expected in 2023).
I see this as more of a concern for counterfactual impact. Where I see it as "StrongMinds got these organizations to do IPT-g, how much better is this than what they'd otherwise be doing?" But maybe I'm thinking about this wrong.