FEM developed a new technology to evaluate our impact: a transmitter that identifies when FEM’s content is on air, and replaces it with music. This means the region surrounding the transmitter is temporarily not exposed to our programmes.
"A similar trend was not seen in Lagos or other Nigerian states, suggesting the change was specific to Kano. We were the only new large scale family planning demand generation activity in the state at that time. We are working with researchers to better understand how much impact can be attributed to FEM’ campaigns. But these results are promising."
I'm interested to hear how you might figure this out. Off the top of my head I struggle to imagine what data might help you drill down more on the cause and effect here, outside of asking women deeper questions. I wonder why PMI didn't just ask the women as well what contributed to them starting to use family planning for the first time, obviously that data would be flawed but if a decent number said "I considered it after hearing on the radio" that would help your case.
If even half of these 250,000 new users (as a small comment it was strange that PMI had no error bars on their graphs) are due to FEM then this must be one of the most staggering results in the history of family planning interventions. I would be interested to hear if you know of any other interventions which have led to even close to that many new family planning users is a year?
The 2023 survey (which just have already been done) will surely shed more light on this as well.
And the RCT design sounds great. My instinct is to be careful about expectations from here, even if you never replicate this first year of apparently outrageous success and only produce a tenth of that impact from here on in, your approach may still be plenty cost effective enough to continue your growth trajectory.
Exciting work! Where can I find details about the survey - methodology and results? What contraception methods were most commonly used and how was use assessed? What was the continuation rate? Was there any impact on unplanned pregnancy rate?
Thanks for the link. So the PMA survey shows a significant increase in modern contraceptive prevalence rate starting in Feb ‘21 - a full 7 months BEFORE the start of the FEM radio campaign - without any appreciable change on slope of the increase after the beginning of the FEM campaign (shaded area on the graph). How are you attributing ANY increase directly due to the FEM campaign itself? And any thoughts about what led to the increased rate preceding the start of the campaign?
I think you're reading the graph wrong. There are two data points, one with data collection "Dec 2020- Feb 2021" (before the intervention) and one with data collection "Dec 2021- Jan 2022" (after the intervention).[1] There are no in-between measures that would give you a "slope" from Feb ‘21 to the start of the intervention.
Thanks. In reviewing the survey methodology more carefully, I see the data comes from cross-sectional survey over 3 months at yearly time points Therefore impossible to say when during the year the increased use began. But as Nick and the post clearly state, if this was the only major intervention during 2021, it’s reasonable to assume that at least some of the increase was due to the FEM campaign.
As far as I can see they are inferring casualty (cautiously) because there are other regions that haven't seen the same drastic uptake in family planning on that year as the region they worked in, and no other major player they know of had a major campaign in that area.
They also said they are engaging with researchers to see how much of the increase in uptake might be due to their work, and said that early results are promising. I struggle to understand what they mean by this and have asked above and hope for a reply.
I would agree that it is dangerous to make any strong claim of casualty from any before and after cross sectional research in such a complex scenario. Because of this nature of the survey I don't think FEM can claim cause and effect here with high certainty.
But assuming the survey is sound (I had a look and it seems solid) then I would say at least there is a good chance that some proportion of the increase in uptake is likely because of FEMs work - but how much they could claim is very difficult to ascertain.
They might have other data they are using sad well which I can't see here though, but will wait for a comment from FEM.
To be clear, I'm a gynecologist. I think increasing contraceptive use is extremely important and impactful and that mass media campaigns are likely to be effective. However, I find some of the claims in this narrative very misleading. I encourage the FEM team to engage in more rigorous research and to focus on capacity-building perhaps by partnering with other organizations that have a great deal of experience working in this space over a long period of time.
Hi Paula, (1) "I encourage the FEM team to engage in more rigorous research and to focus on capacity-building..." FWIW, I think giving the organization (who are engaging with this thread) more time to respond on the point you raise about the PMA survey would be significantly preferable than making a conclusion in either direction at least based on my reading of this thread. (2)"...partnering with other organizations that have a great deal of experience working in this space over a long period of time..." Two separate points: a. to what extent do you know they don't do this? I know them to have extensively engaged with local organizations and had at least information sharing conversations with international organizations (f.e. Development Media International who were founded in 2005) from my very loose following of their work over the years. b. Sadly, and much to my personal heartbreak, from having worked in international women's human rights myself for a number of years, it is certainly not the case that an organization who has been around a long time is more effective or even effective at all than an evidence-based, well run and connected new organization. I have actually been deeply disappointed by the average long-standing organization in this field but perhaps you have had better luck! :) Thanks!
Fair enough. Was going mostly from what I've read in their post and on their website in addition to the RP report. I don't disagree. Although, some international organizations partnering with local public health departments in LMICs have a decent track record in this regard.
Impressive! All of it is a success--the simplicity of the idea, the good implementation, the partnerships with institutions while this topic is far from being culturally obvious...Well done! Truly impressive!
I was incredibly impressed by the tables of numbers in their impact evaluation. After conversing with the team, I've witnessed their high ability to produce results, and their evaluation research methods certainly attest to this. This appears to be one of those rare opportunities where donations could have a significant counterfactual impact.
Edit: I am not in any way affiliated with FEM and randomly met one of the co-founders on a flight where we had a conversation about their work.
Do you have a sense of a) why the Nigerian government’s ministry of health hasn’t prioritised this previously, b) whether they could eventually co-fund / take over these programs?
There have been some government programmes raising awareness about reproductive health and contraception. They have been limited due to fund limitations. There has been no precedent of the government of Nigeria taking over NGO high-intensity media campaigns as far as we are aware, and we expect to be a good partner for the government even at scale.
Wow, you guys are doing so great. I'm curious how you do the creative content? How do you find the people and manage the process using local writers and actors? I would also love to see any information on why you do radio instead of say TikTok?
This is the third in a sequence of posts taken from my recent report: Why Did Environmentalism Become Partisan?
Summary
Rising partisanship did not make environmentalism more popular or politically effective. Instead, it saw flat or falling overall public opinion, fewer major legislative achievements, and fluctuating executive actions.
Public Opinion...
I think right now EAs might be making a significant mistake by paying insufficient attention to the political realm. As EAs we tend to figure out what’s most impactful for us to work on and focus hard. That’s great! But there are various actions that are ‘non-delegatable’ - the extent to which an individual can do the action is limited (like voting, going to a protest, making hard money contributions to particular campaigns). It might be useful if we were all more in the habit of doing variou...
New Video from AI in Context: The Fall and Rise of Sam Altman
If you want to skip straight to the video, here it is!
AI in Context is excited to be back with our fourth video! For those just hearing from us, we make videos for 80,000 Hours, telling stories about transformative AI...
∼250,000 new contraceptive users, ∼20 million new listeners and up to ∼60x cost effectiveness
TL;DR:
In 2021, FEM launched our pilot: a 3-month family planning radio campaign, educating listeners about maternal health and effective contraception. Our ads and shows played up to 860 times in Kano state in northern Nigeria, reaching ~5.6 million. An independent survey showed that in 11 months, contraceptive use increased by ~75% among all women in the state. Since then, we launched a 9-month campaign, aired proofs of concept in three new locations and re-aired in 8 more states, reaching an estimated 20 million new listeners. We were recommended by Giving What We Can and Founders Pledge, who assess our campaigns as ~22 times as effective as cash transfers. We developed new technologies that will allow us to customise health advice to listeners and run a randomised controlled trial. We continue to build organisational capacity to scale across Nigeria and new cause areas in the years ahead.
Studio host during show recording in Kano
Promising results from our pilot (Oct - Dec 2021)
In Nigeria, 1 in 22 women dies of pregnancy-related complications.
In 2020, this meant ~1,047 mothers died for every 100,000 births. For contrast, in the United Kingdom, there were 9.8 maternal deaths per 100,000 births.
Across Nigeria, about half of women have an unmet need for effective contraception. Increasing access to contraception reduces unwanted pregnancies and increases space between births, making pregnancies much safer. It can save the lives of thousands of women and save others from devastating health conditions such as obstetric fistula, and postpartum anaemia and depression. When women and girls understand and access family planning services, they receive substantial positive effects: get more education, make more money, and enjoy the ability to take better care of their children.
Launched in September 2020, Family Empowerment Media develops evidenced-based radio campaigns to improve maternal and child health by raising awareness about contraception. In our Proof of Concept phase, we chose Kano, partnered with a Nigerian organisation, and conducted a short radio campaign.
In September 2021, on World Contraception Day, we launched a pilot campaign in collaboration with the Nigerian Ministry of Health. We produced radio programmes, including dramas, storytelling shows, Q&A programmes with health experts, and interviews with prominent religious leaders on the topic of childbirth spacing.
Our programmes reached a total of 5.6 million listeners up to 860 times and it looks like they had a substantial impact.
Data from pre-post survey by PMA, intervention period is indicated by the shaded area. Over 11 months, we saw an increase from 10% to 19% among married women and from 8% to 14% among all women.
An external survey on contraceptive use in Kano saw a 75% increase (among all women) between February 2021 and January 2022, corresponding to approximately 250,000 new contraceptive users. This is twice as many new users as over the last 5 years combined.
A similar trend was not seen in Lagos or other Nigerian states, suggesting the change was specific to Kano. We were the only new large scale family planning demand generation activity in the state at that time. We are working with researchers to better understand how much impact can be attributed to FEM’ campaigns. But these results are promising.
New technology to evaluate our impact (Jan -Dec 2022)
FEM is committed to collectingrobust, scientific data about the effects of our work. With funding from Grand Challenges Canada (Stars in Global Health), FEM developed a new technology to evaluate our impact: a transmitter that identifies when FEM’s content is on air, and replaces it with music. This means the region surrounding the transmitter is temporarily not exposed to our programmes.
Transmitter technology being tested in Kano State
With this technology, we can randomise listeners into control and treatment groups, enabling randomised controlled trials of radio interventions in regions where this was previously impossible. We plan to launch an RCT of our intervention in early 2024. We will be assisted by a team of academics with diverse expertise in maternal and child health, mass media, and health systems in Nigeria: Maggie McConnell, John Marshall, Belinda Archibong and Dylan Groves.
Among our other innovations, FEM is testing an AI WhatsApp chatbot to facilitate 1:1 follow-up and receive real-time feedback. Listeners send questions, a large language model generates two potential responses, and a trained human operator picks one or writes a custom response. We are testing this as a part of developing our data pipeline. We hope listener feedback will help us maintain an up-to-date understanding of how our material is received. Further, FEM can address popular and generalisable questions in subsequent programs. The WhatsApp source is but one of many datastreams.
Launching our first 9 month campaign (May 2022 - Feb 2023)
In May 2022, we launched a nine-month campaign in Kano state, to an audience base of 5.6 million over 2600 times. In this campaign, we produced a serial drama for the first time – an educational programme lasting eight weeks, following the stories of three families.
These stories were received favourably by men and women in our target population. For this work, our team, including our partner iDevPro-African Institute, received an Award for Excellence from the National Youth Progressive Council, an umbrella organisation for youth (ages 15-35) NGOs and youth organisations across Nigeria.
Throughout our first two years we conducted research with 2,000 individuals, hundreds of hours of desk research, and experimented with different implementation models, landing on one we believe can scale. We also improved on the systems we use to manage campaign development processes.
Our Kano team receives an Award for Excellence from the Youth Progressive Council on the right
Scaling our work: three new proofs of concept (Oct - Dec 2022)
From October to December, we completed proofs of concept in three new Nigerian states: Ondo, Kogi, and Anambra. Here, our programmes reached an estimated 10 million additional listeners. (Our reach data for these locations is less robust than in Kano; we expect to have better estimates in the future.) For proofs of concept, we collaborated with our Nigerian partners to:
Hold a stakeholder workshop with over 40 attendees, including Ministry of Health, religious leaders and civil society organisations
Hold two focus groups with target listeners
Develop 2 ads, based on desk research and focus groups
Work with radio station to broadcast campaign in the state for one week
Survey 40 target listeners, to assess if people have heard and understood our content
We conducted these proofs of concept to test core region-specific assumptions, including partner reliability and stakeholder buy-in. Testing these early prevents waste as we scale.
Despite a flood earlier in the year in Kogi and Anambra, our partners were able to implement the plan. We moved forward with two of these organisations to our formative research phase this year.
Focus group discussion from Anambra
Cost-effectiveness estimates
Charity evaluator Founders Pledge started recommending FEM and offered us a grant from their Global Health & Development Fund in 2022. In a research report on mass media interventions (summarised here), they estimated FEM’s work in northern Nigeria as 22 times as effective as unconditional cash transfers, when considering the impact on maternal health. Based on Founders Pledge’s assessment, Giving What We Can, a nonprofit devoted to highly effective giving, now recommends FEM as a top-rated charity.
Our work also seems promising according to other methods of evaluation. If we enter our reach and cost into a model developed by Givewell, we find our intervention 27 times as effective as cash transfers in the north of Nigeria. In contrast to the models above, Givewell’s model includes more impacts than those on maternal health. It could be that it underestimates our impact on maternal deaths because it is not adjusted for the difference in maternal mortality rates in northern Nigeria vs. other places on which the model was based; Kano has a nearly 4x higher maternal mortality rate than Burkina Faso, for example. This adjustment (which we will make when GiveWell publishes its broken-down moral weights for increased family planning uptake) would likely reveal that our intervention is more effective.
Rethink Priorities recently published this research project on our work suggesting we may be as much as 60 times as cost-effective in northern Nigeria as unrestricted cash transfers. This is highly encouraging.
Meet the FEM team
Since our last update, FEM registered as a 501(c)3 in the US, an Incorporated Trustee in Nigeria and a CIC in the UK, developed strong financial management systems, and made some amazing hires. Our team now consists of 10 ambitious, impact-driven individuals including the heads of our partner organisations:
Anna Christina Thorsheim - Executive Director & Co-Founder - co-founded a health-tech startup developing VR-based phobia treatment and worked as Head of PR at a startup incubator for for-profits aiming to achieve Sustainable Development Goals.
Tolulope Oyekanmi - Head of Growth - has 10+ years experience implementing and advising reproductive, maternal, and child health projects and worked with the Clinton Health Access Initiative and Population Services International across northern and southern Nigeria.
Lia Boldt - Director of Programmes - worked in Cambodia, Ethiopia, Germany, and South Africa, supporting education and health projects at UNESCO and GIZ, and co-founded the nonprofit Sahakom e.V. to enhance global education.
Kabiru Abdulkareem - Research and Process Manager - managed field activities at New Incentives, a GiveWell recommended organisation, and supported eHealth Africa in implementing GIS information technology programmes to monitor polio immunisation campaigns.
Dr. Tukur Mansur Muhammad - Implementing Partner, northern Nigeria - founded and directs iDevPro-African Institute, has served as a Regional Manager for Marie Stopes in northern Nigeria and become one of Nigeria’s foremost family planning experts, with a PhD in international health.
Dr. Charles Umeh - Implementing Partner, southern Nigeria - director of Parkers Mobile Clinic, public health physician and telemedicine expert, member of the International Medical Informatics association and Healthcare Informatics Society of Nigeria.
Dr. Ireti Adesida - Implementing Partner, western Nigeria - founder and director of Rhealyz Global Empowerment Initiative ‘Rhealyz Africa,’ also founded Food Gist Hub, an agricultural communication project, and HeForShe.
Dr Usman Isyaku - Head of Research - has 10+ years experience advising research projects and training students in Nigeria; he investigated climate change mitigation and adaptation strategies at the University of Bergen and consulted for the World Bank’s Development Impact Evaluation Research Group.
Rowan Lund - Operations Manager - supported the operations, community engagement, and public relations of various charities, including Animals Aotearoa and Stop Ecocide.
Harmony Hart - Communications Manager - has 10+ years experience in the nonprofit world as a board president, writer, editor, and movement builder, and has founded and cofounded multiple small for-profit businesses.
Other important partners include UNITRONIC (Pavel Strnad, Zuzana Strnadova, Josef Liska) and Freedom Radio (Ibrahim-Jamilu Dahiru).
Key learnings
Partnering with Nigerian organisations is highly valuable - We’ve had the opportunity to collaborate with four Nigerian organisations, which has greatly increased our effectiveness, and been mutually enriching. FEM shares capacity and extensive research, and offers training on how to produce evidence-based social and behavioural change campaigns. Our partners provide a wealth of knowledge about Nigeria, operational skills, and help managing stakeholder relationships. They also identify and engage trusted religious leaders, community heads, celebrities and health professionals whom we can spotlight in our radio content.
We can re-air content across states - One of our core objectives for 2023 was to test the capacity of our implementing partners to air content in states similar to those where they currently operate. Our partners are based in Kano, Kogi, Ondo, and Anambra. Over the past 6 months, we have also aired content in 7 new states – Enugu, Abia, Delta, Osun, Ogun, Oyo, and Niger – reaching an estimated total of 20-25 million (incl. partners’ states). This successful project aired our content 630 more times and established that our partners can implement regionally not just in their home states.
Adjacent interventions also seem cost-effective - We compared the projected cost-effectiveness of family planning programs in other countries to producing radio programmes on different, adjacent issues in Nigeria (including exclusive breast feeding, intimate partner violence, and kangaroo mother care). We can use the knowledge, connections, and infrastructure we’ve already built to produce campaigns in these new areas. We are now more open to expanding across causes, in addition to scaling up our family planning work.
Our biggest mistakes
We hired too slowly - FEM depended on a small number of people for a long time. After hiring in 2022 and 2023, we have more stable capacity. We also kept applicants waiting a long time through the hiring process. Since we have resolved this challenge and have more capacity, we expect all candidates to hear back in 1-2 weeks per stage.
We didn’t keep you updated - Our blog was last updated a year and a half ago. We’ve heard the question, “Is FEM still active?” The answer is “Yes! More than ever!” We hope to prioritise keeping you more frequently updated.
We had some methodological mistakes in our survey - We selected survey participants based on a criteria that fluctuated between baseline and endline surveys, making it difficult for us to be sure we were comparing similar groups. Though we saw a slight increase in contraceptive use in this data too, we don't trust it that much. In our next roll-out we will work with an external survey firm to evaluate our programmes. We believe the PMA data above is the most reliable data source, indicating our impact.
Our biggest successes
We built a great product and strong processes - We believe our product is evidence-based, context specific, and truly empowering. Our content goes through 4 rounds of reviews, including with the Ministry of Health, and our prototypes have received excellent feedback. Our research suggests our content enables access to informed decision-making about births and birth spacing.
We secured buy-in and built partnerships - FEM collaborates with a wide range of stakeholders, from Nigerian state Ministries of Health and local religious leaders to engineering and tech institutions. We spotted impactful opportunities, and maintained these collaborations.
We hired great people and developed a supportive, cohesive culture - A fast moving organisation, we work together in an encouraging and appreciative way, and reward when people notice their mistakes.
What we’re doing now
We are enabling more women and men to reach their reproductive goals by closing information gaps and spreading the knowledge needed to make informed choices. As we enter the second half of 2023, here’s how we’re doing that:
Preparing for the RCT - We are continuing to prepare for the randomised controlled trial of our intervention in three regions in Nigeria, with which we’ll reach an estimated 10-15 million listeners. We estimate this campaign will lead to ∼147,000 new contraceptive users, save ∼345 lives, and avert ∼5,000 unsafe abortions. It will also generate important, robust data about our intervention. We are conducting formative research and producing content across these new regions this year. We have also signed Memorandums of Understanding, formal collaboration agreements, with the state Ministry of Health in Anambra, Ondo and the Primary Health Care Agency in Ondo.
Re-airing - Meanwhile, we will re-air our radio content with a different station partner in Kano, to create additional impact in northern Nigeria. We chose Arewa station, which reaches 4.6 million Nigerians, including over 1 million mostly rural women of reproductive age. We expect to avert ∼70 maternal deaths through this campaign.
Our plan for the years ahead
Our three-year plan is ambitious but achievable and could catalyse major improvements in global health and wellbeing.
We anticipate launching our RCT in March 2024.
While running the RCT, we will prepare for multiple outcomes by conducting one pilot in a new cause area, and one family planning pilot in a new country.
Our selection of a new country and intervention will follow the same rigorous process that led us to choose Nigeria and family planning. This includes evidence reviews, cost effectiveness modelling and expert consultation. At this time, it looks particularly promising to work in campaigns covering topics related to perinatal health (exclusive breastfeeding, Kangaroo Mother Care, ANC visits) and child health (malaria, pneumonia and diarrhoea). With regard to countries, we are optimistic about Chad, Niger and DRC. Decisions will be made by November 2023, after additional research and expert consultation.
Completing these three projects by 2026 will enable an evidence-based decision about how FEM can make the greatest positive impact on lives, health, and empowerment. If RCT findings support our current estimates of impact, we will have already overcome many hurdles associated with working in new countries. We will re-air content from our RCT to all cost-effective Nigerian states, potentially reaching 35 million listeners for 23 months.
Above, you can see a map of Nigeria – the colours indicate the states that we can reach in collaboration with different partners (each colour corresponds with a partner). The states in black are too dangerous for us to work.
Our vision as an organisation is to save and improve the maximum number of lives, cost-effectively, at scale. We aim to reach 50 million people by 2030, with cost-effective campaigns across a range of high-impact areas. Radio programmes are cheap to produce. Once made, they can be aired again and again. For relatively little money, we can transform people’s lives for the better by increasing their access to life-changing information.
And you can help!
Support our work
Donate. We can reach 5 million listeners for a minute with $14, and you can sponsor a week’s airtime costs for ads for $980. Our 2023 gap is $342k. You can reach out to [email protected] for additional donation options.
Focus group discussion from Anambra
This is SO COOL!
Really amazing progress all around 👏