Comments
If you are one of the 166 million Americans who wear glasses, take them off. Now imagine going your whole life like that. Not fun, right? Whether due to prohibitive cost or a lack of awareness/education, over a billion people are estimated to be living with avoidable vision impairment.
Sounds like a promising cause area! GiveWell has done a few analyses on eyeglasses provision and is funding an ongoing RCT that will provide our best evidence yet on the effects of correcting vision impairment. They’ve concluded the evidence is stronger for productivity effects than for educational effects; the RCT will focus on productivity. In this post, I’ll review their glasses -> productivity cost-effectiveness analysis (CEA), summarizing what GiveWell found and adding some new wrinkles. In a subsequent post, I’ll take a stab at an education-focused CEA, which GiveWell has not yet done.
GiveWell’s Evaluations
GiveWell completed their “Eyeglasses for workers” CEA in 2024, producing two cost-effectiveness estimates from two different RCTs. GiveWell measures the cost-effectiveness of an intervention relative to the cost-effectiveness of simply giving people money, and they set a "bar" of how many times better an intervention must be in order to be funded. Both of these estimates (15x, 11x) cleared the bar in 2024 (10x), and clear the current bar (6x) by an even wider margin. This promise, mixed with uncertainty about how well the results would generalize beyond these studies, led GiveWell to fund a large RCT in Kenya and India that will read out results over the next few years.
If you’ve never looked through a GiveWell CEA, this might be a good place to start! It’s on the simpler side, so it's more approachable than most, and yet the number of necessary assumptions going into the model will likely surprise you.
Let’s summarize the two studies.
- In rural Bangladesh, 423 people with presbyopia (basically age-related far sightedness) were given glasses. Around half of them worked in “vision-intensive occupations” (ex. tailors, carpenters, factory workers)
- GiveWell finds a 21% increase in average incomes (after outlier adjustment)
- They then make two key adjustments that lower the treatment effect:
- #1: Validity concerns with self-reported income, plus potential bias from researchers with a stake in a positive result, lower the effect to 13%
- #2: The effect is cut in half to reflect how a wider-scale intervention would be carried out, where a much higher proportion of those given glasses would not be in a “vision-intensive occupation”.
- A few smaller adjustments for glasses replacement, distribution logistics, seasonality, and trial-vs-real-world effect sizes bring the final estimate to a 5% increase in income
- This income gain, estimated to last for two years (the average lifespan of these glasses), leads to an ROI that is 15x above GiveWell’s benchmark
- 376 tea pickers with presbyopia in Assam (a Northeast Indian state) were given glasses
- Productivity increased 22% in the treatment group compared to the control group over the 11-week study period
- GiveWell estimates this led to a 5% increase in income, due to Assam's wage structure passing less than half of productivity gains through to workers wages
- After small adjustments (the same categories as THRIVE above), they end up with a 3% increase in income
- This income gain, estimated to last for two years (the average lifespan of these glasses), leads to an ROI that is 11x above GiveWell’s benchmark
In GiveWell’s words:
“We estimate that the provision of eyeglasses for working age adults with near-vision impairments may be above our cost-effectiveness bar. However, we currently have a number of uncertainties around our estimate, such as the effect of the program when it is not targeted to specific occupations, the generalizability of the existing evidence for GiveWell’s grantmaking, and the lifespan of the eyeglasses.”
Enter EARN. While the studies above each gave a few hundred people glasses, the “Economic Advantages of Readers for Near vision” (EARN) trial will give 10,000 people glasses! With such a large sample size and two locations (India and Kenya), the study will be able to assess differential effects by location and occupation, and will have richer longitudinal data (check-ins at 12 and 24 months) than the CEA RCTs.
Most of the assumptions in the CEA seem very reasonable to me, and I love EARN as a next step. However, there are a few questions I have, and some additional research that may be of interest.
Questions
- 1. Given GiveWell’s recent cash influx, does it make sense to wait several years for EARN results before funding more interventions?
This seems like a promising area, and funding the best organizations doesn’t need to wait until the RCT results are in. It’s possible the $3.8 million GiveWell has invested in this program is the appropriate amount to spend on an uncertain “near-bar” program, but with an estimated $1 billion going out the door this year, I wonder if it would be worth funding and beginning to scale some promising orgs. Clarity on how GiveWell thinks about the wait-vs-fund tradeoff would be helpful.
- 2. The CEA assumes an average lifespan of two years for each pair of glasses. Is this too low?
GiveWell found this number from a no-longer-present quote on VisionSpring’s website: “VisionSpring calculates the annual increase in earning potential for each pair of glasses sold and multiplies that by the two-year lifespan of a pair of glasses”. It's not clear if this number is data-driven or just a ballpark estimate for their marketing materials.
I was able to find one concrete piece of data. VisionSpring found that 93% of paying customers were still wearing glasses 5-6 years later, with 61% purchasing replacement pairs. That implies that 32% of customers were still wearing their original glasses 5-6 years later. If we assume an exponential decay, 32% of people wearing their original glasses 5.5 years later implies an average lifespan of 4.8 years.
That number is probably an overestimate for two reasons. One, exponential decay is probably not quite right and people are more likely to stop using their glasses earlier on. Two, paying customers are probably more likely to use their glasses longer. Still, a lifespan of 3 years would increase the program’s ROI by 50%.
This is one of the three main uncertainties mentioned in GiveWell's own writeup, so it’ll be interesting to see what they find.
- 3. The CEA assumes incomes revert back to what they would have been in the no-glasses counterfactual after two years. That makes sense if glasses are prohibitively expensive. But are they?
No! An Indian study found that only 31% of people with presbyopia hadn’t bought glasses due to cost. Another study in India found 17.5% of people didn’t buy glasses for cost reasons. Many of those who didn’t cite cost as the main factor cited a lack of need or a lack of awareness.
Most people can afford the ~$10 cost (especially after their incomes have increased). How should we adjust the CEA to account for this?
Let’s assume that 50-75% of people in the study will notice their income rise and attribute that to glasses, and half of those people will start purchasing glasses after their free ones are unusable. Let’s say the average worker has 20 years left in their career. We can adjust that down to 15 years, both because some people will stop buying replacement glasses over time, and because some of them would have started buying glasses eventually even without this program (so crediting all future purchases to the intervention would overstate its effect).
These adjustments, which seem conservative, would still increase the program’s ROI by roughly 3x. Persistent effects could be an important missing piece in GiveWell’s CEA.
- 4. The CEA is focused on the ~30% of productivity gains that go to wages. The ~70% of gains that go to business owners could be meaningful. Should that go into the ROI calculations?
GiveWell’s big shift in their assessment of GiveDirectly was due to spillover effects, where money circulates through the local economy. Increased profits to business owners are a trickier case, but there is some reason to expect spillover effects.
The vast majority of the businesses involved in these studies are domestically owned. The business owners are likely too rich for increased profits to increase their consumption and too rich for us to care much about that increased consumption. But their increased savings and investment could have substantive spillover effects.
Increased investment in their own businesses would likely lead to additional jobs and/or higher pay, which could be even more impactful than GiveDirectly’s spillover effects. Other forms of domestic saving and investment would have smaller spillovers (in roughly descending order): investing in local companies, saving in a local bank, saving in a national bank, investing in the domestic stock market.
This is potentially relevant to many of GiveWell’s analyses. It’s a challenging modeling problem, but I’d love to see them take a stab at it. This could plausibly lead to a large increase in the estimated ROI of many interventions. Gathering data on the savings/investment/consumption habits of business owners might be a good place to start.
- 5. Spillover effects from wage gains should also be incorporated! How large might this effect be?
In November 2024, GiveWell raised their GiveDirectly cost-effectiveness estimate 3-4x. They found that cash transfers generate large spillovers as money circulates around the local economy. Recipient consumption accounts for only ~30% of the total treatment effect, with 70% coming from spillovers. The eyeglasses CEA predates this finding.
GiveWell isn't sure how well the GiveDirectly finding generalizes. They suspect that “part of what’s driving the large spillovers of GiveDirectly are entire villages receiving large consumption shocks ~simultaneously, which helps kickstart local economic activity.” They're funding an RCT in Malawi in hopes of getting a better grasp on spillovers, with results expected to roll in over the next couple years.
Even without the large shocks, we should still expect some spillover effects. Incorporating spillovers into the GiveDirectly estimates increased the program’s expected effectiveness by 1.5-2x. Even if spillover effects are more like 1.25x here, that could meaningfully change funding decisions.
New Research
A new RCT came out a few months ago from the PROSPER research group, creatively titled PROSPER II. The new evidence points in the direction of a lower ROI, but also brings up the interesting proposition of employer-funded interventions.
PROSPER II studied how receiving glasses affected the productivity of sewing machine operators. The study found a 5.7% increase in productivity, far below the RCTs in the initial CEA. Applying GiveWell’s THRIVE-based 30% productivity-to-wage rate, we’d only be left with a 1.7% increase in income. Still, with some of the adjustments I propose in Question 3, that would clear GiveWell’s new 6x bar. Also, the post-intervention period in PROSPER II was only 12 weeks (compared to 8 months in THRIVE), so it's possible the effects would have grown with more follow-up time.
More promisingly, PROSPER II found that the program, if funded by factory owners, would be highly profitable. The factory’s ROI over the 12-week period was over 300%, and if that productivity remained for a full year, the ROI would be over 1500%! The company involved, Shahi Exports, has committed to vision screening for all ~100,000 workers. This suggests that educating and/or subsidizing business owners could be another promising approach here.
Here are a couple inspiring PROSPER II videos:
Non-RCT Research
GiveWell tends to shy away from non-randomized evidence, but there are three other studies worth discussing.
The strongest study comes from Guatemala. Researchers used a difference-in-differences (DiD) design to compare the productivity effects for coffee harvesters given glasses to those without vision impairment. This evidence is not as strong as the RCTs above, since we don’t have randomization, but DiD analyses can make much stronger causal claims than cross-sectional or pre-post analyses. They found an 8% increase in productivity, right in line with the other research we’ve discussed.
Another study in Bangladesh compared vision impaired workers to non-vision impaired workers. This study was cross-sectional, and the inability to rule out confounding means it probably doesn’t belong in a CEA. They found vision impaired workers made roughly 10% less than non-impaired workers, which is in line with the other research.
Lastly, a South African study did a pre-post analysis of textile workers given glasses. They found a 6% increase in productivity. This may be an overestimate since the pre-post design can’t account for “learning-by-doing” improvements. In the PROSPER study, the control group’s productivity substantially increased in the post period. In THRIVE, it did not. Given that counterfactual uncertainty, we shouldn’t put much weight on these findings.
Taken together, these studies don't amount to strong enough evidence for updating the CEA, but they do provide some reassurance that the productivity effects in the RCTs are reasonable.
Conclusion
GiveWell’s CEA seems reasonable to me and I’m very excited to see the results of the EARN RCT. If I were to adjust the CEA, I’d focus on three updates:
- Include the benefits that persist after the free glasses are no longer usable. This would roughly double the ROI
- Reduce the expected treatment effects by 30-50%
- Include a modest spillover effect from the wage increases
The first two revisions would more or less cancel each other out. The third would leave us with a slightly higher estimated treatment effect than the initial CEA, but with plenty of uncertainty that EARN will hopefully help resolve. I’d also advocate for research on the effects of productivity gains that don’t go to wages and what happens to people once their free glasses are no longer usable.
There is a lot riding on getting this right and it would be big news if these intervention are effective. GiveWell’s analysis estimated that there are over 100 million potential recipients of eyeglasses for presbyopia, which means there would be room for over $250M in annual funding.
