Deconstructing the AI Ecosystem: How Do We Isolate Component CEA within a Multi-Modal Child Wellbeing App?
Executive Summary
I recently won funding to develop a multi-modal, AI-enabled application for Community Health Workers (CHWs) in rural Zambia. The app captures voice narrations in local languages and uses computer vision to assist with diagnosis and multi-sectoral triage across Health, Nutrition, and Child Protection.
While traditional NGOs love integrated programming, it presents a classic challenge for EA-style scrutiny: causal dilution.
I am new to the EA community. My background is in ICT and monitoring, evaluation, research, and learning (MERL) in Zambia, rather than pure academic economics. I am writing this post to introduce myself and to solicit feedback on how to mathematically disaggregate and evaluate the cost-effectiveness of a single automated path within a complex, multi-sectoral digital ecosystem.
The Context: Field ICT/MERL Background Challenged by EA Rigor
By way of introduction, I manage MERL for a portfolio of 33 grants worth $25.7M at World Vision Zambia. My career has centered on building data infrastructure to track massive interventions across health, nutrition, education, social protection and child protection, including:
Zambia Multiple Micronutrient Supplementation (ZMMS): Building the data ecosystem for the ongoing implementation research project.
World Food Programme (WFP): Developed a management information system tracking all strategic outcomes and a mobile app for tracking beneficiaries of COVID-response social cash transfers.
Zambia National Malaria Elimination Programme (NMEP): Developed tracking systems for the ITN mass distribution campaigns, alongside an app monitoring the distribution of Against Malaria Foundation (AMF)-supplied ITNs across six provinces.
While I have co-authored a paper on WASH (2025) and a malaria poster for ASTMH (2022), because of my interest in environmental science, my core skillset has generally centered on data and database management, field logistics, and grant compliance.
However, the little I've learnt about EA so far has challenged my thinking. Traditional development focuses heavily on coverage, integrated interventions, and sustainability. But I have recently learnt that EA focuses heavily on marginal cost-effectiveness and unbundled, single-variable impact. I would like to deepen skills in bridging that gap using my new project as a case study.
How the app works: A multi-modal AI core
To mitigate low literacy and take the guesswork out of CHW diagnostics, the app processes unstructured, multi-modal data in real-time at the edge:
Audio narration and NLP: The CHW narrates the interaction in their local language (e.g., "I am with Betty, a 2-year-old girl who has been vomiting since yesterday... she has a bruise on her left elbow and her feet seem swollen"). The app uses Natural Language Processing to translate, parse keywords, and prompt the CHW to run specific tests (like an RDT for malaria or checking for pitting edema).
Computer vision and clinical automation: The app uses the phone camera to interpret Mid-Upper Arm Circumference (MUAC) readings, analyze video for chest in-drawing or fast-breathing (pneumonia tracking), verify pitting edema via the 3-second rule (SAM detection), and capture patterned bruises to determine if a child suffered physical abuse.
Dual-layer logic: It generates a baseline opinion using hardcoded clinical parameters, paired with a second opinion driven by Retrieval-Augmented Generation (RAG) mapped directly to standard WHO and Zambian Ministry of Health treatment guidelines.
Automated multi-sectoral triage:
Health: Severe malaria (and other cases that CHWs are not allowed to treat such as pregnant women) alerts instantly update health facility databases so clinic staff are prepared for the incoming referral.
Nutrition: Growth-faltering triggers immediate, localized nutrition protocols.
Protection: Suspected physical abuse text alerts are sent anonymously to Social Welfare and the record of the suspected case is erased from the CHW's device.
WASH: An algorithmic trigger flags clusters (e.g., if 5 cases of diarrhea occur within 5 days in one locality) and alerts WASH teams to potential water-point contamination.
To operate seamlessly in far-flung, off-grid Zambian villages, the app is built as an offline-first system. Core clinical logic and WHO/ Ministry of Health diagnostic parameters are entirely hardcoded on the device to prevent latency. The system queues unstructured audio and multimodal inputs locally, executing vector store queries (via Vertex AI) for complex edge cases, and pushing batched data packages to the centralized dashboard only when network connectivity is available.
Technical Puzzles I Need Help Solving
Because the app dynamically triggers distinct pathways from a single audio string, calculating standard Cost-Effectiveness Analyses becomes highly complex. I would love the forum's thoughts on three specific methodological hurdles:
Puzzle A: Shared Cost Allocation in Multi-Modal Architecture
The cost of building the core LLM pipelines, audio transcription models, and RAG architecture is shared equally across every module.
The Question: When calculating a CEA specifically for the Malaria referral component or the Nutrition component, how should these fixed technical overhead costs be apportioned? Should they be split strictly by the volume of algorithmic triggers tripped (e.g., if 60% of prompts end up being malaria-related, does malaria absorb 60% of development costs)? What is the EA standard for cost-accounting in shared AI infrastructure?
Puzzle B: Causal Attribution of Automated Cross-Pollination
Let's say a CHW opens the app strictly because of a child protection concern, like a bruise. During the voice narration, the AI picks up on an auxiliary symptom or the camera catches a low MUAC reading. The app prompts a nutrition intervention, and the child's health status subsequently improves.
The Question: The user's primary intent was protection, but the health outcome was nutritional. How can we rigorously structure telemetry data to assign a causal weight to an outcome when an AI core actively forces cross-pollination across distinct sectors?
Puzzle C: Commensurability (Child Protection/WASH vs. DALYs)
Quantifying the cost-effectiveness of malaria interventions or micronutrients may be straightforward because we can map them directly to DALYs or mortality reductions. On the other hand, to me it seems automated social welfare alerts or automated WASH cluster triggers are kind of trickier to convert into standard EA units of impact.
The Question: How do EA researchers handle multi-sectoral programs where one arm produces clear DALY metrics but another arm produces preventive protection or clean-water system metrics? Do you try to convert structural alerts into DALY equivalents, or do you evaluate them on entirely separate tracks?
Proposed Evaluation Ideas (Please feel free to critique)
To isolate impact for future scaling, here are a few ideas I am considering to ground-truth the data. Please tear them apart:
A/B Feature Gating: We randomly assign rural zones into different cohorts. Cohort 1 gets the app with only the core health/malaria pathways active. Cohort 2 gets the fully integrated health, nutrition, WASH, and protection pipelines. We then measure the marginal delta in specific health outcomes between the cohorts over 12 months. Is this logistically ethical and statistically viable for an early-stage digital tool?
Counterfactual Mapping via NMEP/Government Baselines: Utilizing my background with the Zambia National Malaria Elimination Programme data ecosystems, we map our app's specific malaria and pneumonia referral success rates against neighboring control districts that rely purely on traditional, non-digital paper triage.
Moving Forward
I am deeply fascinated by the level of rigor the EA community brings to global health. While my day job involves managing large, complex portfolios where integrated programming is the administrative norm, I want to train myself to think at the marginal, unbundled level.
I would highly appreciate any reading recommendations, existing CEA frameworks for multi-sectoral AI products, or blunt critiques of my proposed methodologies.
Note: If you want to look at some of my past field data infrastructure work, I have linked my co-authored 2025 WASH paper and 2022 ASTMH malaria poster on my personal website: wong-zulu.guru/portfolio.
