All of Derek's Comments + Replies

As you note, this was written in 2012. Have you looked for more recent research into lie detection? As well as fMRI and polygraph, there's voice stress analysis, non-verbal cues, microexpressions, and cognitive interviewing. I did a brief search a while ago but couldn't find anything particularly accurate. I would be extremely keen to hear about any methods with high sensitivity and/or specificity, or with the potential to achieve that in the near future. I might be willing to pay someone a modest amount to review the evidence and predict when accurate tec... (read more)

DALYs appear to weight pain very lightly. For example, terminal illness with constant, untreated pain has a disability (DALY) weight of 0.569, which is only 0.029 more than the weight for the same condition with pain medication. QALYs are better at capturing pain: physical pain is the dimension given the highest weight in the EQ-5D, and instrument used to measure quality of life.

You might want to check disability weights for other painful conditions; I don't remember if they were generally low. 

I suspect QALYs still underweight extreme pain, for vario... (read more)

Thanks for writing this - there's some good stuff here. A few comments: 

1 QALY is equal to a year of life in full health, while 0 QALYs is a health state equivalent to death … The QALY scale admits scores below zero, which represent states worse than death

Minor point, but I think 'being dead' is more accurate than 'death'. The latter suggest permanency, whereas values <0 can represent temporary states that are deemed worse than being dead. That said, there is some uncertainty over the meaning of negative valuations, and the best interpretation may ... (read more)

The urgency of relieving severe physical pain reveals the serious limitations of the “QALYs gained” approach to measuring scale of impact. A person with terminal cancer treated with morphine for two months might remain highly disabled and in a very poor state of health, and gain only a fraction of a QALY, yet be spared two months of agony.

I'm not sure I follow this. QALYs allow negative values, so if morphine treatment increased health-related quality of life from, say, -0.5 to +0.1, it would gain 0.6 QALYs per year. Most/all currently-used value sets... (read more)

Note that there are also methods for calculating confidence intervals around ICERs that avoid issues with ratios. The best I'm aware of is by Hatswell et al. I have an Excel sheet with all the macros etc set up if you want.

MAICER = maximum acceptable incremental cost-effectiveness ratio. This is often called the willingness to pay for a unit of outcome, though the concepts are a little different. It is typically represented by lambda. 

The CE plane is also useful as it indicates which quadrant the samples are in, i.e. NE = more effective but more costly (the most common), SE = more effective and cheaper (dominant), NW  = less effective and more costly (dominated), and SW =  less effective and cheaper. When there are samples in more than one quadrant, which is v... (read more)

This is a recognised issue in health technology assessment. The most common solution is to first plot the incremental costs and effects on a cost-effectiveness plane to get a sense of the distributions:

Then to represent uncertainty in terms of the probability that an intervention is cost-effective at different cost-effectiveness thresholds (e.g. 20k and 30k per QALY). On the CEP above this is the proportion of samples below the respective lines, but it's generally better represented by cost-effectiveness acceptability curves (CEACs), as below:

Often, ... (read more)

For traditional QALY calculations, researchers simply ask people how they feel when experiencing certain things (like a particular surgery or a disease) and normalize/aggregate those responses to get a scale where 0 quality is as good as death, 1 is perfect health, and negative numbers can be used for experiences worse than death.

This isn't correct. QALY weights are typically based on hypothetical preferences, not experiences.

What Richard described is more like a WELBY, which has a similar structure but covers wellbeing in some sense rather than just... (read more)

Glad you found it useful. I am not qualified to comment on the role of neuron count in sentience; you may want to look at work by Jason Schukraft and others at Rethink Priorities on animal sentience and/or get in touch with them.

If you haven't already, you may also want to review the 2018 Humane Slaughter Association report, which was the best I could find in early 2019. While looking for it, I also just came across one from Compassion in World Farming, which I don't think I've read.

On fish, there were several comments here, including this one from me.

The 2018 Humane Slaughter Association report was probably the best info available at the time; not sure what's happened since. 

There are also easy-access savings accounts giving a bit more than 1.3%: https://www.moneysavingexpert.com/savings/savings-accounts-best-interest/

There is a lot of potential in fish welfare/stunning. In addition to what others have mentioned, IIRC from some reading a few years ago:

•  The greatest bottleneck in humane slaughter is research, e.g. determining parameters/designing machines for stunning each major species, as they differ so much. There just aren't many experts in this field, and the leading researchers are mostly very busy (and pretty old), but perhaps financial incentives would persuade some people with the right sort of background to go into this area.
• As well as electrical and percu

Yeah that's what I use, and it's cheaper than the fancy Wiley-branded fish-based product he linked to.  You can get much cheaper fish oil, but if you're going to get the expensive stuff anyway (I guess due to concerns about the quality of the cheaper brands), why not get vegan?

FYI the E-QALY work has been progressing quite well since you asked that question; I've just come out of a webinar on it. Let me know if you want me to send you notes/slides.  

A few key points:

1. The measure has been named the EuroQol Health and Wellbeing (EQ-HWB); E-QALY seems to be what they are calling the broader project of extending the scope of the QALY.
2. Psychometric work and stakeholder consultation resulting in a 25-item 'long' measure, then further consultation resulted in a 9-item EQ-HWB-S (Short Form) covering 9 domains: Mobility, Daily activit

Thanks. I tried 5-HTP a few years ago and didn't notice any benefit, but maybe I'll give it another go.

Thanks for the reply. I don't have much more time to think about this at the moment, but some quick thoughts:

1. On time discounting: It might have been reasonable to omit discounting in this case for the reasons you suggest, but (a) it limits comparability across analyses if you or others do it elsewhere; (b) for various reasons, it would be good to have some estimate of the absolute, not just relative, costs and effects of these interventions; and (c) it's pretty easy to implement in most software, e.g. Excel and R (maybe less so in Guesstimate), so there is

There is much to be admired in this report, and I don't find it intuitively implausible that mental health interventions are several times more cost-effective than cash transfers in terms of wellbeing (which I also agree is probably what matters most). That said, I have several concerns/questions about certain aspects of the methodology, most of which have already been raised by others. Here are just a few of them, in roughly ascending order of importance:

1. Outcomes should be time-discounted, for at least two reasons. First, to account for uncertainty as to

Is the CO2 accumulation entirely due to human (or I suppose animal) respiration? So it will typically be worse in small houses with lots of people (holding other factors, like ventilation, constant)?

In a modern house, with no open fires, lead paint etc, what "household air pollution" might there be?

Thanks - this is useful and I will explore some of the suggestions.

Is there much research comparing immediate vs extended release melatonin? E.g.:

1. Is IR better for speeding sleep onset, as one might expect?
2. Does XR actually improve sleep maintenance/duration more than IR? 
3. Do they have the same effect on sleep efficiency?
4. Is the optimal dose the same for each?
5. Dose aside, do combined IR/XR supplements, or taking a bit of each, give you the 'best of both worlds'?

[Edited on 19 Nov 2021: I removed links to my models and report, as I was asked to do so.]

Just to clarify, our (Derek Foster's/Rethink Priorities') estimated Effect Size of ~0.01–0.02 DALYs averted per paying user assumes a counterfactual of no treatment for anxiety. It is misleading to estimate total DALYs averted without taking into account the proportion of users who would have sought other treatment, such as a different app, and the relative effectiveness of that treatment. 

In our Main Model, these inputs are named "Relative impact of Alternative ... (read more)

[Edited on 19 Nov 2021: I was asked to remove the links.]

For those who are interested, here is the write-up of my per-user impact estimate (which was based in part on statistical analyses by David Moss): [removed]

The Main Model in Guesstimate is here: [removed]

The Effect Size model, which feeds into the Main Model, is here: [removed]

I was asked to compare it to GiveDirectly donations, so results are expressed as such. Here is the top-level summary:

Our analysis suggests that, compared to doing nothing to relieve anxiety, MindEase causes about as much benefi

Done: https://forum.effectivealtruism.org/s/2nMw7ASQNQ35iAz4T

Hi Sam,

Thanks for the comments.

1. Have you done much stakeholder engagement? No. I discuss this a little bit in this section of Part 2, but I basically just suggest that people look into this and come up with a strategy before spending a huge amount of time on the research. I do know of academics who would may be able to advise on this, e.g. people who have developed previous metrics in consultation with NICE etc, but they’re busy and I suspect they wouldn’t want to invest a lot of time into efforts outside academia.

I think they’d reject the assumption tha... (read more)

Interesting, thanks!

Thanks Bob! I will probably do this after publishing the next post.

I've made a few edits to address some of these issues, e.g.:

Clearly, there are many possible “wellbeing approaches” to economic evaluation and population health summary, defined both by the unit of value (hedonic states, preferences, objective lists, SWB) and by how they aggregate those units when calculating total value. Indeed, welfarism can be understood as a specific form of desire theory combined with a maximising principle (i.e., simple additive aggregation); and extra-welfarism, in some forms, is just an objective list theory plus equity (i.e., no

Hi Michael. Thanks for the feedback.

A few general points to begin with:

1. I think it’s generally fine to use terminology any way you like as long as you’re clear about what you mean.
2. In this piece I was summarising debates in health economics, and my framing reflects that literature.
3. The main objective of these posts is to highlight particular issues that may deserve further attention from researchers, and sometimes that has to come at the expense of conceptual rigour (or at least I couldn’t think of a way to avoid that tradeoff). Like you, my natural incli

As far as I can tell, it isn't possible to have line breaks in footnotes (though I may just be doing something wrong). This also precludes bulleted/numbered lists, block quotes, etc. Any chance that could be changed? 

H3s are still being converted to regular Paragraph format when I paste them in from GDocs. What am I doing wrong?

I'm sure there are many giving opportunities in global health that are better than the GiveWell top charities, and I'm pleased to see promising small or medium-sized projects like this being brought to the attention of EAs. 

However, I think you should try to get better estimates of QALYs gained (or DALYs averted)—especially if you're going to feature the cost-effectiveness ratio so prominently in your write-up. This should be possible by referring to the relevant literature. The current estimates don't seem all that plausible to me, e.g. an episode of... (read more)

''Next" and "Previous" arrows/buttons at the bottom of a post, to move to the next/previous post - useful when you haven't read the forum for a while and want to catch up. This would obviously have to assume a certain ordering (e.g. chronological vs karma) and selection (e.g. all or excluding Community/Questions), which could perhaps be adjusted in Settings.

Level 3 headings should be supported. Unless it's changed recently, it currently jumps from Level 2 to Level 4, which makes it hard to logically format complex documents.

Thanks for the comments!

1. The put could cover ~90% of the cost of the accelerated production, taking into account the additional costs.

2. Sales are likely to be higher if they move more quickly: the company with the first billion vaccines is likely to sell a lot more items than the company with the second, and this could more than offset any additional costs. (The second may not sell any, even if it’s a good product, if the first can meet all needs quickly enough.)

3. Some variants outlined in the brief, such as declining payouts, can further incent... (read more)

"The longtermist case for animal welfare"

Have you seen this? https://forum.effectivealtruism.org/posts/W5AGTHm4pTd6TeEP3/should-longtermists-mostly-think-about-animals

Suicide is a very poor indicator of the dead/neutral point, for a host of reasons.

A few small, preliminary surveys I've seen place it around 2/10, though it ranges from about 0.5 to 6 depending on whom and how you ask.

(I share your concerns in parentheses, and am doing some work along these lines - it's been sidelined in part due to covid projects.)

Hah! I was working on them before getting sidelined with covid stuff.

I can send you the drafts if you send me a PM. The content is >80% done (I've decided to add more, so the % complete has dropped) but they need reorganising into ~10 manageable posts rather than 3 massive ones.

Thanks Aidan! Hope you're feeling better now.

Most of your comments sound about right.

On retention rates: Your general methods seem to make sense, since one would expect gradual tapering off of benefits, but your inputs seem even more optimistic than I originally thought.

I'm not sure Strong Minds is a great benchmark for retention rates, partly because of the stark differences in context (rural Uganda vs UK cities), and partly because IIRC there were a number of issues with SM's study, e.g. a non-randomised allocation and evidence of social ... (read more)

Do you think GiveWell top charities are the best of all current giving opportunities? If so, what is the next best opportunity?

Do you think adopting subjective wellbeing as your primary focus would materially affect your recommendations?

In particular:

(a) Would using SWB as the primary outcome measure in your cost-effectiveness analysis change the rank ordering of your current top charities in terms of estimated cost-effectiveness?

(b) If it did, would that affect the ranking of your recommendations?

(c) Would it likely cause any of your current top charities to no longer be recommended?

(d) Would it likely cause the introduction of other charities (such as ones focused on mental health) into your top charity list?

How likely is it that GiveWell will ultimately (e.g. over a 100-year or 10,000-year period) do more harm than good? If that happens, what is the most likely explanation?

A recent post on this forum (one of the most upvoted of all time) argued that "randomista" development projects like GiveWell's top charities are probably less cost-effective than projects to promote economic growth. Do you have any thoughts on this?

I like your general approach to this evaluation, especially:

• the use of formal Bayesian updating from a prior derived in part from evidence for related programmes
• transparent manual discounting of the effect size based on particular concerns about the direct study
• acknowledgement of most of the important limitations of your analysis and of the RCT on which it was based
• careful consideration of factors beyond the cost-effectiveness estimate.

I'd like to see more of this kind of medium-depth evaluation in EA.

I don't have time at the moment for a close ... (read more)

There is also evidence that health problems have a much smaller effect on subjective well-being than one might imagine.

This is only the case for (some) physical health problems, especially those associated with reduced mobility. People tend to underestimate the SWB impact of (at least some) mental health problems. See e.g. Gilbert & Wilson, 2000; De Wit et al., 2000; Dolan & Kahneman, 2007; Dolan 2008; Pyne et al., 2009; Karimi et al., 2017

You might want to mention the publication date (1937)

Thanks - I missed that on my skim. But the "extended" follow-up is only for another two months. It does seem to indicate that effects persist for at least that period, without any trend towards baseline, which is promising (though without a control group the counterfactual is impossible to establish with confidence). I wonder why they didn't continue to collect data beyond this period.

Thanks - "trained facilitator" might be a bit misleading. Still, it looks like there were two volunteer course leaders for each course, selected in part for their unspecified "skills", who were given "on-going guidance and support" to facilitate the sessions, and who have to arrange a venue etc themselves, then go through a follow-up process when it's over. So it's not a trivial amount of overhead for an average of 13 participants.