FYI the E-QALY work has been progressing quite well since you asked that question; I've just come out of a webinar on it. Let me know if you want me to send you notes/slides.
A few key points:
Thanks. I tried 5-HTP a few years ago and didn't notice any benefit, but maybe I'll give it another go.
Thanks for the reply. I don't have much more time to think about this at the moment, but some quick thoughts:
This paper, the Drummond book above, and this book are good starting points if you want to learn how to do cost-effectiveness analysis (including sensitivity analysis).
A couple nitpicks:
There is much to be admired in this report, and I don't find it intuitively implausible that mental health interventions are several times more cost-effective than cash transfers in terms of wellbeing (which I also agree is probably what matters most). That said, I have several concerns/questions about certain aspects of the methodology, most of which have already been raised by others. Here are just a few of them, in roughly ascending order of importance:
Topic 2.2: (Re-)prioritising causes and interventions
[…]
GiveWell
[…]
Spillover effects
Secondly, there are also potential issues with ‘spillover effects’ of increased consumption, i.e. the impact on people other than the beneficiaries. This is particularly relevant to GiveDirectly, which provides unconditional cash transfers; but consumption is also, according to GiveWell’s model, the key outcome of deworming (Deworm the World, Sightsavers, the END Fund) and vitamin A supplementation (Hellen Keller International). Evidence from multiple contexts suggests that, to some extent, the psychological benefits of wealth are relative: increasing one person’s income improves their SWB, but this is at least partly offset by decreases in the SWB of others in the community, particularly on measures of life satisfaction (e.g. Clark, 2017). If increasing overall wellbeing is the ultimate aim, it seems important to factor these ‘side-effects’ into the cost-effectiveness analysis.
As usual, GiveWell provides a sensible discussion of the relevant evidence. However, it is somewhat out of date and does not fully report the findings most relevant to SWB, so I’ve provided a summary of wellbeing outcomes from the four most relevant papers in Appendix 2.1. In brief:
As GiveWell notes, it is hard to aggregate the evidence on spillovers (psychological and otherwise) because of:
Like GiveWell, I suspect the adverse happiness spillovers from GiveDirectly’s current program are fairly small. In order of importance, these are the three main reasons:
In addition, any psychological harm seems to be primarily to life satisfaction rather than hedonic states. As noted in Haushofer, Reisinger, & Shapiro (2019): “This result is intuitive: the wealth of one’s neighbors may plausibly affect one’s overall assessment of life, but have little effect on how many positive emotional experiences one encounters in everyday life. This result complements existing distinctions between these different facets of well-being, e.g. the finding that hedonic well-being has a “satiation point” in income, whereas evaluative well-being may not (Kahneman and Deaton, 2010).” This is reassuring for those of us who tend to think feelings ultimately matter more than cognitive evaluations.
Nevertheless, I’m not extremely confident in the net wellbeing impact of GiveDirectly.
A few more notes on interpreting the wellbeing effects of GiveDirectly:
In addition, I would note that the other income-boosting charities reviewed by GiveWell could potentially cause negative psychological spillovers. According to GiveWell’s model, the primary benefit of deworming and vitamin A supplementation is increased earnings later in life, yet no adjustment is made for any adverse effects this could have on other members of the community. As far as I can tell, the issue has not been discussed at all. Perhaps this is because these more ‘natural’ boosts to consumption are considered less likely to impinge on neighbours’ wellbeing than windfalls such as large cash transfers. But I’d like to see this justified using the available evidence.
I make some brief suggestions for improving assessment of psychological spillover effects in the “potential solutions” subsection below.
Four studies investigated psychological impacts of GiveDirectly transfers. Two of these found wellbeing gains for cash recipients (“treatment effects”) and only null or positive psychological spillovers:
However, two studies are more concerning:
Note: GiveWell’s review of an earlier version of the paper reports a “statistically significant negative effect on an index of psychological well-being that is larger than the short-term positive effect that the study finds for receiving a transfer, but the negative effect becomes smaller and non-statistically significant when including data from the full 15 months of follow-up… The authors interpret these results as implying that cash transfers have a negative effect on well-being that fades over time.” I’m not sure why the authors removed those analyses from the final version.
Is the CO2 accumulation entirely due to human (or I suppose animal) respiration? So it will typically be worse in small houses with lots of people (holding other factors, like ventilation, constant)?
In a modern house, with no open fires, lead paint etc, what "household air pollution" might there be?
Thanks - this is useful and I will explore some of the suggestions.
Is there much research comparing immediate vs extended release melatonin? E.g.:
[Edited on 19 Nov 2021: I removed links to my models and report, as I was asked to do so.]
Just to clarify, our (Derek Foster's/Rethink Priorities') estimated Effect Size of ~0.01–0.02 DALYs averted per paying user assumes a counterfactual of no treatment for anxiety. It is misleading to estimate total DALYs averted without taking into account the proportion of users who would have sought other treatment, such as a different app, and the relative effectiveness of that treatment.
In our Main Model, these inputs are named "Relative impact of Alternative App" and "Proportion of users who would have used Alternative App". The former is by default set at 1, because the other leading apps seem(ed) likely to be at least as effective as Mind Ease, though we didn't look at them in depth independently of Hauke. The second defaults to 0; I suppose this was to get an upper bound of effectiveness, and because of the absence of relevant data, though I don't recall my thought process at the time. (If it's set to 1, the counterfactual impact is of course 0.)
Our summary, copied in a previous comment, also stresses that the estimate is per paying user. I don't remember exactly why, but our report says:
Other elements of the MindEase evaluation (i.e. parts not done by Rethink Priorities) consider a “user” to be a paying user, i.e. someone who has downloaded the app and purchased a monthly or annual plan. For consistency, we will adopt the same definition. (Note that this is a very important assumption, as the average effect size and retention is likely to be many times smaller for those who merely download or install the app.)
As far as I can tell (correct me if I'm wrong), your "Robust, uncertainty-adjusted DALYs averted per user" figure is essentially my theoretical upper-bound estimate with no adjustments for realistic counterfactuals. It seems likely (though I have no evidence as such) that:
So 0.02 DALYs averted per user seems to me like an extremely optimistic average effect size, based on the information we had around the middle of last year.
[Edited on 19 Nov 2021: I was asked to remove the links.]
For those who are interested, here is the write-up of my per-user impact estimate (which was based in part on statistical analyses by David Moss): [removed]
The Main Model in Guesstimate is here: [removed]
The Effect Size model, which feeds into the Main Model, is here: [removed]
I was asked to compare it to GiveDirectly donations, so results are expressed as such. Here is the top-level summary:
Our analysis suggests that, compared to doing nothing to relieve anxiety, MindEase causes about as much benefit per paying user as donating $40 (90% confidence interval: $10 to $140) to GiveDirectly. We suspect that other leading apps are similarly effective (perhaps more so), in which case most of the value of MindEase will come from reaching people who would not have accessed alternative treatment.
Due to time constraints and lack of high-quality information, the analysis involved a lot of guesswork and simplifying assumptions. Of the parameters included in our Main Model, the results are most sensitive to the effect sizes of both MindEase and GiveDirectly, the retention of those effects over time, and the choice of outcome metric (DALYs vs WELLBYs). One large, independent study could eliminate much of this uncertainty. Additional factors worth considering include indirect effects (e.g. economic productivity, meat consumption, evidence generation), opportunity costs of team members’ time, and robustness to non-utilitarian worldviews.
Note that this was done around June 2020 so there may be better information on MindEase's effectiveness by now. Also, I think the Happier Lives Institute has since done a more thorough analysis of the wellbeing impact of GiveDirectly, which could potentially be used to update the estimate.
[Recording of the talk and related papers]
You can now view the recording of the talk from Professor John Brazier - Extending the QALY beyond health - the EQ HWB (Health and Wellbeing)
Kaltura
https://digitalmedia.sheffield.ac.uk/media/t/1_8k5slrc4
YouTube
https://www.youtube.com/watch?v=KTlsIvqyhNI
Papers associated with this talk
Special issue of Value in Health Development papers:
Brazier, J et al. ‘The EQ-HWB: overview of the development of a measure of health and well-being and key results’. Value in Health. https://www.sciencedirect.com/science/article/pii/S1098301522000833
Mukuria, C et al. "Qualitative Review on Domains of Quality of Life Important for Patients, Social Care Users, and Informal Carers to Inform the Development of the EQ Health and Wellbeing." Value in Health (2022).
https://www.sciencedirect.com/science/article/pii/S1098301521032277
Carlton, J et al. "Generation, Selection, and Face Validation of Items for a New Generic Measure of Quality of Life: The EQ Health and Wellbeing." Value in Health (2022). https://www.sciencedirect.com/science/article/pii/S1098301522000109
Peasgood, T et al. "Developing a New Generic Health and Wellbeing Measure: Psychometric Survey Results for the EQ Health and Wellbeing." Value in Health (2022). https://www.sciencedirect.com/science/article/pii/S1098301521031922
International papers:
Monteiro AL, et al. A Comparison of a Preliminary Version of the EQ Health and Wellbeing Short and the 5-Level Version EQ-5D. Value Health. 2022 Mar 8:S1098-3015(22)00051-1. doi: 10.1016/j.jval.2022.01.003. Epub ahead of print. PMID: 35279371.
Augustovski F, Argento F, Rocío R, Luz G, Mukuria C, Belizán M. The Development of a New International Generic Measure (EQ Health and Wellbeing): Face Validity And Psychometric Stages In Argentina. https://www.sciencedirect.com/science/article/abs/pii/S1098301522000134