We did a conversion from Excel to R in 2025, although not LLM assisted. It took about three months, which I would say was a disappointing but realistic downside-case timeline (we basically budgeted for two months and something complex came up). So if you're finding you can get it done in weeks rather than months with an LLM you're actually a long way ahead of the state of the art.
One of the big recurring arguments in pharma cost effectiveness modelling is whether to use databases and scripts with a proper statistical programming languages like R, or whether to stick with Excel / Sheets for our models. The advantages of proper languages are manifold, including - as you've pointed out - that you can probably use LLMs on them more successfully to audit and augment your code. However the advantage of spreadsheets is that they are extremely portable, meaning almost anyone can run them natively on their laptop and anyone can understand how they work if they want to change parameters. This matters a lot if transparency is a goal, and transparency is often such an overriding goal that we pick Excel over technically stronger languages. So I'd caution that in converting the existing models to databases / scripts you're actually making implicit decisions about the nature and audience of the model far beyond just whether they are legible to LLMs or not.
I mention this because I'm about to get nerd-sniped by the problem and want to make sure the limitations of the approach are well understood before I get sucked into it!
Really happy you found the critique valuable. I agree with everywhere you've said the issue is too small to be worth addressing, and also with the correction that because AMF and SMC used different data sources it was legitimate not to explicitly model indirect deaths for the AMF (I'm also pleased you've standardised your methodology in the updated model, because this was quite a big discrepancy between approaches).
I don't totally agree with the other areas where you've said you're confident there is no error at all. I think a better interpretation might be that you've got a reasonable simplifying assumption that isn't worth adjusting. For example, you say that it is appropriate to use different methodologies to account for the spillover effects of cash transfer vs income increase from deworming, which results in larger effective household sizes for cash transfer than for deworming. This is because deworming increases income via improving the earning potential of one member of the household, so there needs to be an adjustment for multi-earner households. Obviously you're the topic area experts so I'd defer to you on whether or not this is reasonable, but the way you've implemented this seems contestable to me in the abstract; around 40% of the benefits of GiveDirectly are assumed to be accrued in the form of dividends on a long-term capital investment, but those benefits are distributed across the entire family of 4.7 for the entire ten-year period of the intervention, whereas in the deworming models you explicitly downweight family size to account for dependents leaving the household over time and the possibility of having two earners in a household. It seems more defensible to me to use a constant family size over the two interventions, and instead adjust the number of consumption doublings the deworming intervention provides to account for some families having two wage earners (so a doubling in individual income from the intervention doesn't translate to a doubling in household income in the model). That is to say; either approach seems completely reasonable in isolation, but using a different approach in the two interventions favours GiveDirectly (because a given transfer spread across more people results in more utility overall, due to utility being a function of log income increase). In hindsight it is probably a bit harsh to describe this as an 'error' by GiveWell because it is a good-faith effort to capture the most important part of the model dynamics, but I also don't think it is unambiguously correct as you've indicated in your review document; there's a genuine inconsistency here. Overall I feel the same about most of the other areas you've highlighted - I was probably a bit over-harsh describing them as 'errors', but nevertheless they are potentially risky methodological steps which seem to have simpler and less risky alternative implementations.
Also on a small point of clarification - your comments in the linked document suggest that you are not worried about the hard-coded cells because they don't have any impact on calculations. Sorry if I wasn't clear in the text, but the main impact of hard coded cells is that the model becomes really hard to maintain - you won't have an intuitive sense about whether you can make certain changes or not, because certain calculations you might expect to take place actually won't happen because of the hard coding.
That's really interesting, and honestly pretty surprising - I'd really have to quite radically change my view if it turns out Freedom Farms have found a way to raise >5000 animals on one farm in conditions which are broadly acceptable. If I understand you correctly you're saying that each individual farm could plausibly be much smaller than 5000 animals though, which I would still find interesting that there's a way for the system to produce meat in aggregate without atrocity-level cruelty, but less challenging to my existing worldview because I think it is the 'factory' element of factory farms which is what drives them to be especially cruel.
I'd be very interested in anything you can find on the distribution of farm sizes - or if you can wait a week or two for me to get some work deadlines out the way I'd also be happy to investigate myself and get back to you.
I'm a bit confused. It answers your question unless you believe there are farms with more than half a million chickens / 5000 pigs under farm at a time which are not 'Factory' farms. Do you believe such farms exist? Do you have any evidence they exist? If not, in what way has your question not been answered?
Yes; the Environmental Protection Agency uses various criteria to distinguish between 'Animal Feeding Operations' (AFOs) and 'Concentrated Animal Feeding Operations' (CAFOs, aka Factory Farms). Within this, there is further subdivision between small, medium and large CAFOs. The definition of a 'large CAFO' relies exclusively on the number of animals in that AFO so you can confidently identify a 'large CAFO' using the Sentience Institute methodology. This will undercount the true number of factory farms, since it will miss eg some 'Medium CAFOs' which need to be a certain size AND meet some other criteria about how they handle sewage, but since most factory farmed animals are farmed in Large CAFOs it doesn't make much difference.
To be clear the Sentience Institute itself is beyond reproach, describing their approach as being "We estimate that 99% of US farmed animals are living in factory farms at present", which is totally unambiguous.
I'm quite sympathetic to the idea of moral arguments treating the basic unit of 'meat' as being the animal - that seems to be the morally relevant unit
I researched this fairly extensively a few years ago, and it is a true (but maybe misleading, depending on context) claim.
The usual source for this claim is the Sentience Institute, although if you go to a huge amount of effort to check government records by hand you get basically the same number so I'm not worried that the source is somewhat biased. They get the 99% number by using USDA data on the size of farms, and then defining any farm over a certain size as a 'factory' farm. This makes sense to me, and is how I'd approach the definitional problem unless I was shown extremely compelling evidence of a farm processing eg 5000 pigs a year using traditional 'mom and pop' techniques.
The reason the claim might be misleading is that it is using 'meat' as a shorthand for 'meat animals' rather than eg 'carcass weight'. Because the vast majority of farmed animals are chickens, and chickens are overwhelmingly factory farmed when farmed, the result of the Sentience Institute methodology is that it appears the overwhelming percentage of farmed animals are factory farmed. In fact, by carcass weight it is 'only' about 90% of meat which is factory farmed.
This could in theory drop a bit lower if you say that the process for factory farming cows is not all that morally relevant for the 2/3 of their life they spend in pastures and hence were very exacting with your definitions (ie maybe for the sake of argument we would say something like "85% of meat-by-weight is farmed in a way that would be extremely distressing for the animal" rather than "99% of meat is factory farmed"), but it is hard to get very much lower than this because pigs and chickens are almost exclusively raised in cramped factory conditions and also make up a great deal of the meat we eat.
Ah sorry, I seem to have slightly misled you. The quote which you attribute to Scott is actually written by me and the co-author of an adversarial collaboration hosted on Scott's old blog. I'm not the author of the Adventist Health Study linked, much that I wish I was!
If you have questions about the statistics in the adversarial collaboration I'd be more than happy to talk through the approach we used. If you have questions about AHS2, by all means let's share the work of finding the answer but I can't promise to be any more help than any other random person you'd pick off the street
Hi Elizabeth, I'm the co-author of the piece linked above. You're absolutely right we chose to focus on the omnivore-vs-vegetarian comparison, for a variety of different reasons. However, AHS-2 does have some comparisons between omnivores to vegans. From the abstract: "the adjusted hazard ratio (HR) for all-cause mortality in all vegetarians combined vs non-vegetarians was 0.88 (95% CI, 0.80–0.97). The adjusted HR for all-cause mortality in vegans was 0.85 (95% CI, 0.73–1.01)". So depending on how strict you are being with statistical significance there's somewhere between a small signal and no signal that veganism is better with respect to all-cause mortality than omnivorism.
I think AHS is the best data we've got on this topic, but I'd be cautious about over-interpreting it. In my mind the biggest criticism is that Adventists are generally more healthy than the typical American (they do a lot more exercise, avoid alcohol and tobacco etc), which leads to extremely pernicious selection bias. For example, it could be that a vegan diet is much healthier than an omnivorous diet if you are the kind of person who spends a lot of time worrying about your health generally, but the risk of getting the wrong nutrients is so high with a vegan diet that it is harmful to people who are not otherwise concerned about their health. So I'm not confident the very slight improvement in overall mortality from switching from a vegetarian diet to a vegan diet can be judged to be a real effect from AHS alone.
On the other hand, I think I would be confident enough in the AHS data to say that it shows that veg*nism does not entail a tradeoff on the 'years of life lived' axis. The most conservative reading of the data possible would be that a veg*n diet has no effect on years of life lived, and I think it is probably more reasonable to read the AHS study as likely underestimating the benefits a veg*n diet would give the average person. Obviously 'years of life lived' is not the same thing as 'health' so I'm not saying this is a knock-down argument against your main point - just wanted to contextualise how we were using the data in the linked piece.
We did a conversion from Excel to R in 2025, although not LLM assisted. It took about three months, which I would say was a disappointing but realistic downside-case timeline (we basically budgeted for two months and something complex came up). So if you're finding you can get it done in weeks rather than months with an LLM you're actually a long way ahead of the state of the art.
One of the big recurring arguments in pharma cost effectiveness modelling is whether to use databases and scripts with a proper statistical programming languages like R, or whether to stick with Excel / Sheets for our models. The advantages of proper languages are manifold, including - as you've pointed out - that you can probably use LLMs on them more successfully to audit and augment your code. However the advantage of spreadsheets is that they are extremely portable, meaning almost anyone can run them natively on their laptop and anyone can understand how they work if they want to change parameters. This matters a lot if transparency is a goal, and transparency is often such an overriding goal that we pick Excel over technically stronger languages. So I'd caution that in converting the existing models to databases / scripts you're actually making implicit decisions about the nature and audience of the model far beyond just whether they are legible to LLMs or not.
I mention this because I'm about to get nerd-sniped by the problem and want to make sure the limitations of the approach are well understood before I get sucked into it!
Really happy you found the critique valuable. I agree with everywhere you've said the issue is too small to be worth addressing, and also with the correction that because AMF and SMC used different data sources it was legitimate not to explicitly model indirect deaths for the AMF (I'm also pleased you've standardised your methodology in the updated model, because this was quite a big discrepancy between approaches).
I don't totally agree with the other areas where you've said you're confident there is no error at all. I think a better interpretation might be that you've got a reasonable simplifying assumption that isn't worth adjusting. For example, you say that it is appropriate to use different methodologies to account for the spillover effects of cash transfer vs income increase from deworming, which results in larger effective household sizes for cash transfer than for deworming. This is because deworming increases income via improving the earning potential of one member of the household, so there needs to be an adjustment for multi-earner households. Obviously you're the topic area experts so I'd defer to you on whether or not this is reasonable, but the way you've implemented this seems contestable to me in the abstract; around 40% of the benefits of GiveDirectly are assumed to be accrued in the form of dividends on a long-term capital investment, but those benefits are distributed across the entire family of 4.7 for the entire ten-year period of the intervention, whereas in the deworming models you explicitly downweight family size to account for dependents leaving the household over time and the possibility of having two earners in a household. It seems more defensible to me to use a constant family size over the two interventions, and instead adjust the number of consumption doublings the deworming intervention provides to account for some families having two wage earners (so a doubling in individual income from the intervention doesn't translate to a doubling in household income in the model). That is to say; either approach seems completely reasonable in isolation, but using a different approach in the two interventions favours GiveDirectly (because a given transfer spread across more people results in more utility overall, due to utility being a function of log income increase). In hindsight it is probably a bit harsh to describe this as an 'error' by GiveWell because it is a good-faith effort to capture the most important part of the model dynamics, but I also don't think it is unambiguously correct as you've indicated in your review document; there's a genuine inconsistency here. Overall I feel the same about most of the other areas you've highlighted - I was probably a bit over-harsh describing them as 'errors', but nevertheless they are potentially risky methodological steps which seem to have simpler and less risky alternative implementations.
Also on a small point of clarification - your comments in the linked document suggest that you are not worried about the hard-coded cells because they don't have any impact on calculations. Sorry if I wasn't clear in the text, but the main impact of hard coded cells is that the model becomes really hard to maintain - you won't have an intuitive sense about whether you can make certain changes or not, because certain calculations you might expect to take place actually won't happen because of the hard coding.
That's really interesting, and honestly pretty surprising - I'd really have to quite radically change my view if it turns out Freedom Farms have found a way to raise >5000 animals on one farm in conditions which are broadly acceptable. If I understand you correctly you're saying that each individual farm could plausibly be much smaller than 5000 animals though, which I would still find interesting that there's a way for the system to produce meat in aggregate without atrocity-level cruelty, but less challenging to my existing worldview because I think it is the 'factory' element of factory farms which is what drives them to be especially cruel.
I'd be very interested in anything you can find on the distribution of farm sizes - or if you can wait a week or two for me to get some work deadlines out the way I'd also be happy to investigate myself and get back to you.
I'm a bit confused. It answers your question unless you believe there are farms with more than half a million chickens / 5000 pigs under farm at a time which are not 'Factory' farms. Do you believe such farms exist? Do you have any evidence they exist? If not, in what way has your question not been answered?
Yes; the Environmental Protection Agency uses various criteria to distinguish between 'Animal Feeding Operations' (AFOs) and 'Concentrated Animal Feeding Operations' (CAFOs, aka Factory Farms). Within this, there is further subdivision between small, medium and large CAFOs. The definition of a 'large CAFO' relies exclusively on the number of animals in that AFO so you can confidently identify a 'large CAFO' using the Sentience Institute methodology. This will undercount the true number of factory farms, since it will miss eg some 'Medium CAFOs' which need to be a certain size AND meet some other criteria about how they handle sewage, but since most factory farmed animals are farmed in Large CAFOs it doesn't make much difference.
To be clear the Sentience Institute itself is beyond reproach, describing their approach as being "We estimate that 99% of US farmed animals are living in factory farms at present", which is totally unambiguous.
I'm quite sympathetic to the idea of moral arguments treating the basic unit of 'meat' as being the animal - that seems to be the morally relevant unit
I researched this fairly extensively a few years ago, and it is a true (but maybe misleading, depending on context) claim.
The usual source for this claim is the Sentience Institute, although if you go to a huge amount of effort to check government records by hand you get basically the same number so I'm not worried that the source is somewhat biased. They get the 99% number by using USDA data on the size of farms, and then defining any farm over a certain size as a 'factory' farm. This makes sense to me, and is how I'd approach the definitional problem unless I was shown extremely compelling evidence of a farm processing eg 5000 pigs a year using traditional 'mom and pop' techniques.
The reason the claim might be misleading is that it is using 'meat' as a shorthand for 'meat animals' rather than eg 'carcass weight'. Because the vast majority of farmed animals are chickens, and chickens are overwhelmingly factory farmed when farmed, the result of the Sentience Institute methodology is that it appears the overwhelming percentage of farmed animals are factory farmed. In fact, by carcass weight it is 'only' about 90% of meat which is factory farmed.
This could in theory drop a bit lower if you say that the process for factory farming cows is not all that morally relevant for the 2/3 of their life they spend in pastures and hence were very exacting with your definitions (ie maybe for the sake of argument we would say something like "85% of meat-by-weight is farmed in a way that would be extremely distressing for the animal" rather than "99% of meat is factory farmed"), but it is hard to get very much lower than this because pigs and chickens are almost exclusively raised in cramped factory conditions and also make up a great deal of the meat we eat.
Ah sorry, I seem to have slightly misled you. The quote which you attribute to Scott is actually written by me and the co-author of an adversarial collaboration hosted on Scott's old blog. I'm not the author of the Adventist Health Study linked, much that I wish I was!
If you have questions about the statistics in the adversarial collaboration I'd be more than happy to talk through the approach we used. If you have questions about AHS2, by all means let's share the work of finding the answer but I can't promise to be any more help than any other random person you'd pick off the street
Hi Elizabeth, I'm the co-author of the piece linked above. You're absolutely right we chose to focus on the omnivore-vs-vegetarian comparison, for a variety of different reasons. However, AHS-2 does have some comparisons between omnivores to vegans. From the abstract: "the adjusted hazard ratio (HR) for all-cause mortality in all vegetarians combined vs non-vegetarians was 0.88 (95% CI, 0.80–0.97). The adjusted HR for all-cause mortality in vegans was 0.85 (95% CI, 0.73–1.01)". So depending on how strict you are being with statistical significance there's somewhere between a small signal and no signal that veganism is better with respect to all-cause mortality than omnivorism.
I think AHS is the best data we've got on this topic, but I'd be cautious about over-interpreting it. In my mind the biggest criticism is that Adventists are generally more healthy than the typical American (they do a lot more exercise, avoid alcohol and tobacco etc), which leads to extremely pernicious selection bias. For example, it could be that a vegan diet is much healthier than an omnivorous diet if you are the kind of person who spends a lot of time worrying about your health generally, but the risk of getting the wrong nutrients is so high with a vegan diet that it is harmful to people who are not otherwise concerned about their health. So I'm not confident the very slight improvement in overall mortality from switching from a vegetarian diet to a vegan diet can be judged to be a real effect from AHS alone.
On the other hand, I think I would be confident enough in the AHS data to say that it shows that veg*nism does not entail a tradeoff on the 'years of life lived' axis. The most conservative reading of the data possible would be that a veg*n diet has no effect on years of life lived, and I think it is probably more reasonable to read the AHS study as likely underestimating the benefits a veg*n diet would give the average person. Obviously 'years of life lived' is not the same thing as 'health' so I'm not saying this is a knock-down argument against your main point - just wanted to contextualise how we were using the data in the linked piece.