Hi folks, in this post we’d like to describe our views as the Chair (Peter) and Director (Michael) of HLI in light of the recent conversations around HLI’s work. The purpose of this post is to reflect on HLI’s work and its role within the EA community in response to community member feedback, highlight what we’re doing about it, and engage in further constructive dialogue on how HLI can improve moving forward. 

HLI hasn’t always got things right. Indeed, we think there have been some noteworthy errors (quick note: our goal here isn’t to delve into details but to highlight broad lessons learnt, so this isn’t an exhaustive list):

  • Most importantly, we were overconfident and defensive in communication, particularly around our 2022 giving season post. 
    • We described our recommendation for StrongMinds using language that was too strong: “We’re now in a position to confidently recommend StrongMinds as the most effective way we know of to help other people with your money”. We agree with feedback that this level of confidence was and is not commensurate with the strength of the evidence and the depth of our analysis. 
    • The post’s original title was “Don’t give well, give WELLBYs”. Though this was intended in a playful manner, it was tone-deaf, and we apologise.
  • We made mistakes in our analysis.
    • We made a data entry error. In our meta-analysis, we recorded that Kemp et al. (2009) found a positive effect, but in fact it was a negative effect. This correction reduced our estimated ‘spillover effect’ for psychotherapy (the effect that someone receiving an intervention had on other people) from 53% to 38%  and therefore reduced the total cost-effectiveness estimate from 9.5x cash transfers to 7.5x.
    • We did not include standard diagnostic tests of publication bias. If we had done this, we would have decreased our confidence in the quality of the literature on psychotherapy that we were using. 
  • After receiving feedback about necessary corrections to our cost-effectiveness estimates for psychotherapy and StrongMinds, we failed to update our materials on our website in a timely manner.

As a community, EA prides itself on its commitment to epistemic rigour, and we’re both grateful and glad that folks will speak up to maintain high standards. We have heard these constructive critiques, and we are making changes in response. 

We’d like to give a short outline of what HLI is doing next and has done in order to improve its epistemic health and comms processes.

  1. We’ve added an “Our Blunders” page on the HLI website, which lists the errors and missteps we mentioned above. The goal of this page is to be transparent about our mistakes, and to keep us accountable to making improvements.
  2. We’ve added the following text to the places in our website where we discuss StrongMinds: 
    • “Our current estimation for StrongMinds is that a donation of $1,000 produces 62 WELLBYs (or 7.5 times GiveDirectly cash transfers). See our changelog
      However, we have been working on an update to our analysis since July 2023 and expect to be ready by the end of 2023. This will include using new data and improving our methods. We expect our cost-effectiveness estimate will decrease by about 25% or more – although this is a prediction we are very uncertain about as the analysis is yet to be done. 
      While we expect the cost-effectiveness of StrongMinds will decrease, we think it is unlikely that the cost-effectiveness will be lower than GiveDirectly. Donors may want to wait to make funding decisions until the updated report is finished.” 
  3. We have added more/higher quality controls to our work:
    • Since the initial StrongMinds report, we’ve added Samuel Dupret (researcher) and Dr Ryan Dwyer (senior researcher) to the team, providing more quantitative eyes to double-check all key research inputs and reproduce all key research outputs.
    • We will be more careful about communicating our work, including greater emphasis on the uncertainties in, and the limitations of, our analyses. 
    • In line with this, we’ve developed our process for how to classify the quality of evidence and the depth of our work in a more principled and transparent manner. This will be posted on our website in Q4 2023.
    • We have just added a changelog to the site to keep a public record of how our CEAs have changed over time and why.
    • We have sought advice from leading experts to develop our methods for handling publication bias. We will use these methods in our updated evaluation of psychotherapy, scheduled for Q4 of 2023.
    • We have also further developed our general research methodology to ensure we follow best practices, such as following more closely to recommendations from the Cochrane Collaboration for conducting systematic reviews and meta-analyses. We will post an article outlining our research methodology in Q4 2023.
  4. Updating the StrongMinds analysis in Q4 2023 to include more rigorous methods, the most recent studies, and to address various points raised by feedback from the EA community, including to:
    • Conduct a full systematic review of the evidence, with an improved process for determining the weight we place on different sources of evidence.
    • Update the estimated spillover benefits for household members.
    • Update the cost estimates for StrongMinds.
    • Apply more appropriate checks and corrections for publication bias.
    • Conduct more robustness tests to determine how the effects change under various reasonable analytic approaches.
    • We have addressed the data entry error for spillovers and have updated our analysis with the correct value.
    • We have updated our website to reflect our corrected estimates, and we now note which analyses are being updated.
  5. We are developing new content for our website that will summarise our charity evaluations to date. We produce long technical reports – which we think we need to continue to do  – but we realise that if we only produce these, it can be challenging for others to understand and review our work. We hope these summaries will make it easier for others to understand our evaluations. 
  6. We’ve added an experienced Comms Manager, Lara Watson, to the team. Lara will be helping us improve both our community and academic reporting. We are striving to take a tone in our communications that is more cautious and less adversarial, and Lara will actively double-check for this. 
  7. Since I (Peter) have now stepped down as CEO of Mind Ease and UpLift and handed over the reins, I’ll be resigning from the board to join HLI as a Managing Director in mid-September, where I’ll be providing strategy, comms, and management support to the org. Michael will become Research Director, freeing him up to spend more time on research; moving forward, HLI will have two Co-Directors.

Again, we are grateful to have received feedback from the EA community, and we are looking forward to doing good better. If you’d like to follow HLI’s progress, please sign up for the newsletter in the footer of our site here, and we’ll also be posting updates here on the forums as usual. 

As part of my (Peter’s) new role, I’ll be reaching out to various stakeholders and collecting further feedback on HLI work, what we should do, and what we should do better. If there are any suggestions or concerns you’d like to send to me directly, I’m peter@happierlivesinstitute.org – I’d love to hear from you.

Peter Brietbart (Outgoing Chair of HLI Board of Trustees, Incoming Managing Director)

Michael Plant (Outgoing Director, Incoming Research Director)


 

355

0
0
1

Reactions

0
0
1

More posts like this

Comments16
Sorted by Click to highlight new comments since: Today at 4:15 PM

Thanks for sharing this. I've not been following your work closely, but running a new org with very ambitious goals must be challenging, and I appreciate you acknowledging and sharing your mistakes so far. It would be surprising if you hadn't made a few mistakes at this point. Good luck!

I'm excited about this, thanks Michael and Peter!

This post is why EA is so great. The broader take home is that human life and social systems are infinitely complex and all of EA needs to continue to trend humble regarding our ability to figure things out in the specific window/perspective we have. And since our funding advice goes to the wealthiest and most willing to give humans on earth, we hold a significant sway on which interventions get funded, so if we are off or wrong, we do harm. Thanks for continuing to iterate humbly. 

[comment deleted]6mo2
0
1

Executive summary: The post describes mistakes HLI made in overconfident and inaccurate communications, outlines steps HLI is taking to improve research rigor and communications, and invites further constructive feedback.

Key points:

  1. HLI acknowledges errors like overconfidence in claims about StrongMinds, misleading language, data mistakes in cost-effectiveness estimates, and delayed website updates.
  2. HLI adds transparency with a public "Our Blunders" page and clarity in StrongMinds recommendations.
  3. HLI improves research practices like more reviewer checks, uncertainty communication, and following best practices.
  4. HLI revamps communications with a new Comms Manager role and tone changes.

 

This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.

I've not seen this feature before but it looks awesome! If testing goes well, it would be really cool to integrate it more fully (eg: allow post author's to put this executive summary at the top of their post).

And perhaps test ways of allowing users to edit or correct the summary.

If summaries are editable, it could be nice to keep the same length limit so that they don't balloon during editing.  

Or a character limit. 

edit Also I'm not sure it would balloon. I have runa couple of open documents with 100s or 1000s of editors and my experience is that people obey norms. I guess that if the box said, please be concise, EAs would bel

I'd personally like that, as it summarised only 4 of the 7 points we made.

Thanks for taking costly steps. 

I'm generally of the opinion that it's worth waiting a bit after changes to decide if changes have worked or not, so I'm looking forward to checking back in 6 months. I hope we can become convinced that the situation is unlikely to happen again and the basically not mention this any more.

I am unsure about a general practice of hiring comms people to interact with the community. I guess I'd prefer a frame of mediation - the aim is for us all to communicate better with one another. I softly predict that if every org hired intra-community comms people that would lead to less signal in discussion overall. 

Hey Nathan! 

Appreciate the comment, and totally agreed on your first paragraph. We'll continue to post updates as the work progresses, and we'll welcome feedback and comments as we go.

On the second paragraph, fear not. Lara wasn't brought on to interface with the community in our stead, but provide comms wisdom and support. We've been discussing creating a formal HLI account through which we'd do future posting, but that would still be us at the keys. 

Hi Peter,

We've been discussing creating a formal HLI account through which we'd do future posting, but that would still be us at the keys. 

Would such HLI account be used just for posts, or also for comments? For the latter, I guess personal accounts may be better, because different people from the team can have different views. Aggregating these into a single one would lead to some loss of potentially relevant information.

Yep exactly. Nothing set in stone, but 'official' account for top level posts and then named-person accounts for individual views/commentary seems decent.

Great! Thanks for your work Peter.

Thanks for this thoughtful and grounded post. I feel this shows dedication to improvement and transparency and appreciate that you've openly called out the mistakes HLI has made and identified specific ways you're planning to improve moving forward. These all sound like great steps be taking with direct lines between the missteps and next steps.

PS excited to hear about your new position, Peter! Looking forward to seeing HLI progress under your leadership.