Thanks for the reply Niek. Perhaps I will have time to respond more in depth later but let me just say an important bit:
I said the study and the way it has come about is of questionable integrity. It wasn’t an assessment of the integrity of the people involved.
That being said, we shouldn’t just assume everyone involved in the study to have acted under complete integrity either, given the COI. We have no visibility on the extent to which COI’d authors were involved in editorial decisions during writing, methodological decisions during analysis and data-presentation as well as operational decisions during the study’s actual implementation. In all of these, I’m more concerned about bias (as distinct from lack of integrity), so that’s a good reason to keep matters of personal integrity out of it.
At the end of the day, the Ben et al. study is of low quality and questionable integrity. RP's subjective discounting and limited engagement with the actual validity of the study, its statistical methods and its confidence intervals obscures how unreliable this study seems. Therefore, this independent review from RP should not change much with respect to our confidence in Impala's effect size. Better, actually independent research is needed before we can estimate the effectiveness of the intervention with any certainty.
A few of the issues with the study, beyond what has been raised by RP:
From reviewhistory, it seems that conflict of interest was only declared by version 3, after peer review was done. Reviewers possibly were not aware of the massive conflict of interest from 3 of the study's co-authors, and it is unclear to what extent these co-authors were involved in the study. Peer-reviewers could only see that Niek, the listed co-author, worked at Goal3. It is very possible that they were not aware that Goal3 markets and commercializes the system being studied and therefore didn't register the enormous COI.
The HDU estimate is very fragile. The adjusted 9.8 pp reduction implies that, without IMPALA, these children would have died at well above the 8.1% actually observed before IMPALA, an extrapolation resting on only 20 pre-period deaths. The authors’ own matched analysis gives just 1.7 pp (95% CI −4.3 to 8.4). In the first manuscript version, both the methods and the table labelled this as the effect among children actually monitored. Later versions relabelled it as a whole-cohort comparison without changing the numbers. If the original description is correct, the only estimate for monitored children shows no significant mortality effect.
The patient mix changed significantly from the pre to the post period, and this was accounted for insufficiently. Only a subset of statistically significant difference between the groups are accounted for, without clearly arguing why. They leave out differences in diagnose severity alltogether. It is very hard to perform any analysis of effect size and statistical significance without better information on the causes of mortality and the severity of the diagnoses at entry. RP can't perform primary research, but they can make an effort to request and re-analyze the data for some basic reproduction checks, rather than taking the results and take TMLE-adjusted confidence intervals on face value for statistical tests. The odd TMLE-adjusted mortality effect at the ZSH HDU should flag to the authors that the TMLE method shouldn't be trusted blindly. To me, this seems to be clearly the best use of their time, and I am surprised they didn't attempt to do this more thoroughly.
In ZSH HDU, mortality increases post-impala, but at SLH, we don't have data on mortality and characteristics of the patients that actually were monitored with impala. ZSH seems to be the cleanest evidence of the effect size due to seeming similarities between pre and post groups, but we can't do the analysis without the HDU and non-HDU breakdown.
tl;dr: subjective discounting is not a replacement for critically engaging with evidence quality. The 3x discounting can give the reader the impression that the researchers are being very conservative, but a 12% effect size is still moderately large in absolute terms and large relative to other interventions. It is huge given that most of the effects are from spillover, and mostly covers a reduction in deaths from children not actually monitored by the device. So I warn any readers against accepting the framing that a 0.5-1pp mortality reduction across the entire hospital is a very conservative estimate.
Quick rambly comment from my side (because I didn’t expect this to be spread outside of the publication and wasn’t aware of timelines :))
Most of the writing is done by Dirk-Jan. I gave input and part of this followed from a conversation we had on stage during EAGxAmsterdam.
I think Dirk-Jan is a rare example of an experienced professional from the ODE field who is willing to engage critically with EA ideas in good faith, and I am very happy we started this discussion :) As a community, I think we should cherish encounters with constructive outside experts
My quick take if you need to take one thing away from this thought-piece: we need to make sure that, as a community, we incentivize careful analysis of unintended consequences, and then update accordingly! The incentive is to write away unintended effects qualitatively, or handwave a modest discount (or bonus, for positive effects)
When you start a project/organisation, it makes sense to put most effort into making your theory of change very rigorous and de-risk it as much as possible. Once you scale that project/organisation, you should probably put the same effort on analyzing whether the causal links in your ToC could break-down at scale, and pay attention to the causal effects that fall outside your ToC all together!
The best version of this, as always, employs a good mix of qualitative and quantitative thinking. If anything, I personally think discussions on unintended effects and “backfire risks” are often too qualitative, and miss a serious attempt at sizing.
Lastly, a prompt everyone can use:
What incentives are you creating with this project, especially at scale/maturity?
How will insiders and outsiders perceive your work when you scale it?
And the memeable: “is there anyone you forgot to ask?!” or “who’s left out of this project, but will still experience consequences”
Are you crowding anyone out?
Are there (indirect) costs you are incurring on people?
Hi Andrew, welcome to the forum! We should consider ourselves lucky that you are chiming in! I hope you share your thoughts here, and may you inspire future generations of EA to start big, bold projects in health, poverty reduction and education!
"Are there such opportunities? So far, I can't break this down to anything tangible. 'If we don't do anything, the systems will become entrenched and determine animal outcomes for decades to come' - what systems? What outcomes? Who, where? Can someone give me a few clear examples of tractable situations?
... Agricultural AI platforms from John Deere and Bayer are being deployed. But how is any of this tractable - what are you hoping orgs/grantees/EA people can do about those things?"
... If that's so, then can someone tell me, in plain English, what that looks like? I.e '[lab] is currently planning [this development]. If we do [this action], we can change it to [this outcome], which will mean [x number] of animals experience [less suffering, presumably].'
I just want to express that these are really sound questions and I hope you keep asking them. I believe they are, as of yet, unanswered. I'd love to see AIxAnimals field/thought leaders work out their theories of change more clearly and concretely.
I think you correctly point out that there are gaps, and some of the framings I've seen seem to suggest that poultry farmers will just sit idly by as their animal-friendly AI COO's will make commercially suboptimal farm-management and procurement decisions.
But, this is still a nacent field and I think questions like these will help it along!
It's one of the main factors going into the decisions from most of the funders in the circle :)
However, CEA's are rather shallow and most grants are quite experimental or exploratory in nature sometimes with the intent to allow orgs to scale and find ways to cost-effectively use funds at a larger scale in the future, either by themselves or by others.
Thanks Sofia! You should mainly thank Joey and James, who typed most and set the norm for transparency.
I definitely second that there's room for a guide that helps people seeking funding in the AW space with understanding where and when to apply, based on the scope, size and maturity of their organisation. This might be something the AIM grantmaking team could help with at a point in the future and I know Joey's quite keen on this, but Hive might be in an even better position to work on sth similar :)
Thanks for the reply :) the number you cite for other insects is yearly turnover, not how many are alive at any moment, right? So the number might not be as far off?
But yeah, without looking at any data, other major farmed insects numbers are probably increasing at a faster rate than honeybees.
Holy **** this was one of those things that changed my mind just by looking at the title because I realised I was holding a belief that was clearly mistaken if you think about it logically for a second, but no one actually ever really challenged me on it. Why did I never realise that honeybees are obviously the most farmed animal on earth?
I always told people that farmed honey is terrible, but terrible in the same way as that dairy is terrible. I would have put the order of 'plz stop eating this for the love of god' at
eggs & white meat
red meat,
dairy and honey
I will bump honey up the list (50% vibes based list by the way, not calibrated on exact SAD/portion).
PS: Would have been easier if you listed '#days of farming' per 'per capita consumption' rather than per kilo. 1kg is roughly the yearly consumption per person in Europe I believe (surprisingly high?), vs 20kg for chicken. Also, the link to the 97% thing didn't work, so I couldn't really check that stat. (It seems oddly specific given uncertainty)
Thanks Björn! Really interested in the diet change interventions. Veganuary has probably also had quite a nice spillover effect through corporate engagement with the challenge (wether solicited by veganuary or not). During the month of January, supermarkets and brands tend to allocate more resources on marketing, product development and deals for plant based products.
To me, this write-up would have been a lot more useful if it ended with a rough cost-effectiveness analysis and any takes on what funders should be excited to fund.
Some other thoughts: I'm sceptical that every 6 extra sign ups to a challenge will get someone from an average diet to an almost plant based diet for a duration of more than 2 years. Is that what the 15% suggests? How were the RCT's performed? Were random people selected and given an incentive to participate?
Thanks for the reply Niek. Perhaps I will have time to respond more in depth later but let me just say an important bit:
I said the study and the way it has come about is of questionable integrity. It wasn’t an assessment of the integrity of the people involved.
That being said, we shouldn’t just assume everyone involved in the study to have acted under complete integrity either, given the COI. We have no visibility on the extent to which COI’d authors were involved in editorial decisions during writing, methodological decisions during analysis and data-presentation as well as operational decisions during the study’s actual implementation. In all of these, I’m more concerned about bias (as distinct from lack of integrity), so that’s a good reason to keep matters of personal integrity out of it.
At the end of the day, the Ben et al. study is of low quality and questionable integrity. RP's subjective discounting and limited engagement with the actual validity of the study, its statistical methods and its confidence intervals obscures how unreliable this study seems. Therefore, this independent review from RP should not change much with respect to our confidence in Impala's effect size. Better, actually independent research is needed before we can estimate the effectiveness of the intervention with any certainty.
A few of the issues with the study, beyond what has been raised by RP:
tl;dr: subjective discounting is not a replacement for critically engaging with evidence quality. The 3x discounting can give the reader the impression that the researchers are being very conservative, but a 12% effect size is still moderately large in absolute terms and large relative to other interventions. It is huge given that most of the effects are from spillover, and mostly covers a reduction in deaths from children not actually monitored by the device. So I warn any readers against accepting the framing that a 0.5-1pp mortality reduction across the entire hospital is a very conservative estimate.
Quick rambly comment from my side (because I didn’t expect this to be spread outside of the publication and wasn’t aware of timelines :))
Most of the writing is done by Dirk-Jan. I gave input and part of this followed from a conversation we had on stage during EAGxAmsterdam.
I think Dirk-Jan is a rare example of an experienced professional from the ODE field who is willing to engage critically with EA ideas in good faith, and I am very happy we started this discussion :) As a community, I think we should cherish encounters with constructive outside experts
My quick take if you need to take one thing away from this thought-piece: we need to make sure that, as a community, we incentivize careful analysis of unintended consequences, and then update accordingly! The incentive is to write away unintended effects qualitatively, or handwave a modest discount (or bonus, for positive effects)
When you start a project/organisation, it makes sense to put most effort into making your theory of change very rigorous and de-risk it as much as possible. Once you scale that project/organisation, you should probably put the same effort on analyzing whether the causal links in your ToC could break-down at scale, and pay attention to the causal effects that fall outside your ToC all together!
The best version of this, as always, employs a good mix of qualitative and quantitative thinking. If anything, I personally think discussions on unintended effects and “backfire risks” are often too qualitative, and miss a serious attempt at sizing.
Lastly, a prompt everyone can use:
Hi Andrew, welcome to the forum! We should consider ourselves lucky that you are chiming in! I hope you share your thoughts here, and may you inspire future generations of EA to start big, bold projects in health, poverty reduction and education!
I just want to express that these are really sound questions and I hope you keep asking them. I believe they are, as of yet, unanswered. I'd love to see AIxAnimals field/thought leaders work out their theories of change more clearly and concretely.
I think you correctly point out that there are gaps, and some of the framings I've seen seem to suggest that poultry farmers will just sit idly by as their animal-friendly AI COO's will make commercially suboptimal farm-management and procurement decisions.
But, this is still a nacent field and I think questions like these will help it along!
It's one of the main factors going into the decisions from most of the funders in the circle :)
However, CEA's are rather shallow and most grants are quite experimental or exploratory in nature sometimes with the intent to allow orgs to scale and find ways to cost-effectively use funds at a larger scale in the future, either by themselves or by others.
Thanks Sofia! You should mainly thank Joey and James, who typed most and set the norm for transparency.
I definitely second that there's room for a guide that helps people seeking funding in the AW space with understanding where and when to apply, based on the scope, size and maturity of their organisation. This might be something the AIM grantmaking team could help with at a point in the future and I know Joey's quite keen on this, but Hive might be in an even better position to work on sth similar :)
Thanks for the reply :) the number you cite for other insects is yearly turnover, not how many are alive at any moment, right? So the number might not be as far off?
But yeah, without looking at any data, other major farmed insects numbers are probably increasing at a faster rate than honeybees.
Holy **** this was one of those things that changed my mind just by looking at the title because I realised I was holding a belief that was clearly mistaken if you think about it logically for a second, but no one actually ever really challenged me on it. Why did I never realise that honeybees are obviously the most farmed animal on earth?
I always told people that farmed honey is terrible, but terrible in the same way as that dairy is terrible. I would have put the order of 'plz stop eating this for the love of god' at
I will bump honey up the list (50% vibes based list by the way, not calibrated on exact SAD/portion).
PS: Would have been easier if you listed '#days of farming' per 'per capita consumption' rather than per kilo. 1kg is roughly the yearly consumption per person in Europe I believe (surprisingly high?), vs 20kg for chicken. Also, the link to the 97% thing didn't work, so I couldn't really check that stat. (It seems oddly specific given uncertainty)
Thanks Björn! Really interested in the diet change interventions. Veganuary has probably also had quite a nice spillover effect through corporate engagement with the challenge (wether solicited by veganuary or not). During the month of January, supermarkets and brands tend to allocate more resources on marketing, product development and deals for plant based products.
To me, this write-up would have been a lot more useful if it ended with a rough cost-effectiveness analysis and any takes on what funders should be excited to fund.
Some other thoughts: I'm sceptical that every 6 extra sign ups to a challenge will get someone from an average diet to an almost plant based diet for a duration of more than 2 years. Is that what the 15% suggests? How were the RCT's performed? Were random people selected and given an incentive to participate?