Cross posted from my blog https://hauke.substack.com/p/mindease

Luke Muehlhauser recently published "EA needs consultancies". This is an example of an EA consulting project. If you're interested in consulting services from me, get in touch at: H@EA.do

Acknowledgements

This work was funded by the Total Portfolio Project. Thanks to Derek Foster from Rethink Priorities and the Total Portfolio Project for their help in improving this manuscript. All opinions are mine.

Our process

This report was written in roughly 20 days of desk research in early 2020. The evaluation was partly based on a prototype beta version of Mind Ease and pilot data on the effectiveness of it dating back even further. Other parts of our analysis assume a more idealized and further optimized version of Mind Ease. Mind Ease’s team is constantly working to improve the app and has for instance added an additional intervention since this review was conducted and made lots of other improvements. Since the review, the Mind Ease team reports to have “additional evidence that was not evaluated in this review, in particular: (1) lots of app usage data from our actual users with information about how much they improve, and (2) a randomized controlled trial that we conducted early on in the life of the project that we used to narrow down our set of planned interventions to use in the app.”

Executive Summary

This report evaluates the social impact of Mind Ease, an app to reduce anxiety. The main goal of this report is to estimate the number of disability adjusted life years (DALYs) averted per user. This is to inform a separate analysis on whether to make an impact investment. 

Our key question here was: how large of a counterfactual social impact does Mind Ease have?

To answer this, we tackled the following four related questions:

As a cause area, is anxiety a large and neglected problem?

Are Mind Ease exercises based on evidence-based psychotherapeutic interventions that are effective in an offline context?

Is there evidence from the scientific literature that anxiety can be effectively reduced if these offline psychotherapeutic interventions are ported over to mental health apps?

Is there evidence that Mind Ease in particular is effective in reducing anxiety?

In chapter 1, we briefly consider anxiety as a cause area. In brief, we find that the scale of untreated anxiety is relatively big and neglected. 

We find that more than 1% of all ill-health and death—27 million disability adjusted life years—were caused by anxiety. This is similar to the global burden of violence. Globally, 284 million people—3.8% of all people—have anxiety disorders. Other estimates suggest that this might be even higher: according to the CDC, 11% of U.S. adults report regular feelings of worry, nervousness, or anxiety and ~19% had any anxiety disorder in the past year according to the NIH and Anxiety and Depression Association of America.

Mental health in general has been described as a “truly neglected area of global health policy”. The net present cost of scaling up anxiety treatment to an adequate level from 2016–30 has been estimated to be $56 billion. This could perhaps be seen as the overall value of Mind Ease’s market size that could be “disrupted”. The expected returns of scaled-up treatment would be 6 million extra years of healthy life valued at $52 billion. Given that this is the case, the average cost-effectiveness of anxiety treatment might be quite high already (roughly $10k per QALY gained=$56 billion / 6 million years of healthy life), it is plausible that there are even more highly cost-effective interventions to treat anxiety. Scaled-up treatment might also increase labour productivity by $169 billion, leading to benefit cost ratios 3 (or 4, when the value of health returns is also included). 

In chapter 2, we review the scientific literature on the psychotherapeutic interventions Mind Ease tries to reduce anxiety with (e.g. Cognitive Therapy, Progressive Muscle Relaxation).

On the whole, Mind Ease’s current ten interventions seem carefully selected to be evidence-based. Though the scientific literature in clinical psychology often has poor methodology, many of Mind Ease’s interventions are backed up by relatively substantial evidence from systematic reviews and meta-analyses, and, where those are unavailable, plausible other evidence exists that suggest that the interventions are beneficial. The interventions usually have small to moderate effects in an offline therapeutic context, but these effect sizes might be reduced when the interventions are implemented in an app.

In chapter 3, we review this literature on mobile health (mHealth), a burgeoning field in health where apps are used to treat illness effectively at scale. In particular, we focus on mental mHealth to treat anxiety. We find that there is some emerging evidence that the effects of the interventions reviewed in the previous chapter transfer to an online context and that apps can reduce anxiety, but likely with a smaller effect size. We find that there is some evidence from systematic reviews and meta-analyses that apps, especially evidence-based apps like Mind Ease, can indeed reduce anxiety, but the effect sizes will likely be smaller than in an offline context.

In chapter 4, we evaluate direct evidence for Mind Ease’s efficacy such as user reports, internal self-evaluation, and their internal study. We find that results from internal studies are promising and perhaps hint at Mind Ease being effective to reduce anxiety. However, there are several methodological concerns and only one study outcome is useful for comparing it to other health interventions. We suggest an ideal trial to test Mind Ease’s effectiveness going forward. Taken together with user reports and the evidence reviewed in the previous chapters, it is plausible that the usage of an optimized future version of Mind Ease might significantly reduce anxiety for some people—also with non-trivial effect sizes. We also review the evidence of competitor apps and find they might also be quite effective at reducing anxiety, which could reduce Mind Ease’s counterfactual impact.

Chapter 1-4 served as priors for chapter 5, where we analyse the cost-effectiveness of Mind Ease in terms of quality adjusted life years gained (or disability adjusted life year averted) per additional user. 

In a final literature review we find that, generally, studies often show that mental health apps have very high cost-effectiveness. This is due to their zero marginal cost per user, compared to the high cost of conventional psychotherapy, which is highly-skilled labor intensive. We also review what the size of the quality of life gains / reductions in disability weights of reducing anxiety in other studies are. We use these to feed them into our cost-effectiveness analysis.

Ultimately, we find that the long-term benefits of anxiety reduction of consistent long-term usage of an optimized future version of Mind Ease targeted at moderately anxious populations could conceivably have a benefit of 1 DALY averted per user. However, due to some inherent uncertainty in the data, estimated conservatively it might be as low as 0.002 DALY averted per user or as high as 6.11 DALY averted per user in an optimistic scenario. To counterfactually adjust for the fact that users might find other treatment, through apps or other forms of therapy, our best guess is that Mind Ease counterfactually averts 0.25 DALY per user. However, again due to the inherent uncertainty in the data, the counterfactual might be as low as 0.0001 DALY per user under conservative assumptions, and 4.07 DALY per user averted under optimistic assumptions.

For a crude comparison, the Against Malaria Foundation has a cost-effectiveness of roughly $50 per DALY averted (this figure might be somewhat out of date, but should be roughly correct). This is because a death of an under 5-year-old is equivalent to ~34 Years of Life lost (YLL) per AMF death. One of the most effective global health charities—the Deworm the World Initiative—roughly averts a DALY equivalent for $14, while GiveDirectly, the philanthropic benchmark averts a DALY equivalent for roughly $860. This means that if Mind Ease can reach ~4 users for less than $50 (or $12.5 per user), and avert 0.25 DALYs in each, then it could be as cost-effective as AMF. The promise of mobileHealth (mHealth) is that at scale apps often have ‘zero marginal cost’ per user (much less than $12.50) and so plausibly are very cost-effective. One can calculate the cost-effectiveness of an impact investment differently, and this conservative “societal perspective” includes the cost to the user and assumes a philanthropic subsidy. In contrast, if investing in Mind Ease returns profits above market rate in expectation, if we only look at the cost to the government and the philanthropist, the cost-effectiveness of Mind Ease might even be “negative”, i.e. they might save users, the government, and the philanthropist money.

We close with some qualitative arguments in the form of crucial considerations. For instance, funding Mind Ease might have benefits beyond its direct impact, through the value of improved global mental health and the value of information of researching what works in mental mHealth. We provide some toy models showing that Mind Ease’s research might plausibly have even higher cost-effectiveness than its direct impact if it were to improve the field of anxiety treatment as a whole.

Full Report

Please read the full report here.

Further reading

Founders Pledge Mental Health Cause Area Report

Anxiety apps: Can you lessen anxiety by playing a game on your phone?