This report was conducted within the pilot for Charity Entrepreneurship’s Research Training Program in the fall of 2023 and took around eighty hours to complete (roughly translating into two weeks of work). Please interpret the confidence of the conclusions of this report with those points in mind.

We are grateful to the organisations who took the time to engage with this research. Being transparent about your own monitoring, evaluation and cost-effectiveness is hard and we applaud all efforts that try to share this data openly with people trying to understand the impact that the organisation has. 

For questions about the structure of the report, please reach out to Leonie Falk at leonie@charityentrepreneurship.com. For questions about the content of this research, please contact Moritz Stumpe at moritz@animaladvocacyafrica.org or Aashish Khimasia at aashishkhimasia@gmail.com.

The full report can also be accessed as PDF here.

Executive summary

This executive summary synthesises the findings from our in-depth review of Animal Advocacy Careers (AAC), focusing on their different programmes, effectiveness, and areas for improvement.

AAC plays a critical role in the animal advocacy movement by addressing talent bottlenecks. Organisations within the movement highly value the access to qualified candidates, a gap AAC aims to fill through its various services, including career advice, workshops, and job boards.

AAC's theory of change (ToC) is robust and predominantly supported by evidence. It suggests a positive impact on animal welfare through its programmes. However, this review identifies key uncertainties that need addressing. Firstly, the potential negative externalities of AAC's services, such as inadvertently directing individuals towards less optimal career paths or inducing anxiety over career choices, require careful monitoring and research. Secondly, the efficacy of AAC's recruitment services, despite high demand, remains a point of scepticism. These services need close evaluation to determine their future role in AAC's portfolio.

A standout aspect of AAC's methodology is the innovative use of Importance- and Counterfactual-Adjusted Placements (ICAPs) for evaluating its impact. This approach indicates a positive cost-benefit ratio, with AAC generating approximately USD 2.6 for every dollar spent. However, this metric's robustness could be further enhanced. Recommendations include adopting systematic approaches such as confidence intervals and developing a detailed rubric for counterfactual judgement. We also recommend that AAC collect stronger data for backing some of the assumptions in its monetary valuation of one ICAP. Such refinements would increase the accuracy and transparency of AAC's impact assessments.

Regarding organisational health, AAC demonstrates stability and efficiency with no major concerns. Their transparency and leadership's clear future vision are commendable. However, some improvements can be made to further strengthen their organisational framework.

Finally, we advise AAC to try to diversify their funding sources. Their current primary reliance on a few large donors within the Effective Altruism community might present a risk.

In summary, AAC stands out as a valuable contributor to the animal advocacy movement. It effectively addresses crucial talent needs and demonstrates impactful programme execution. However, there is room for enhancement in its monitoring systems, recruitment service, and funding diversification. Overall, our assessment of AAC is positive, with targeted recommendations aimed at maximising its impact and ensuring its long-term success in the movement.

1     Background

“Animal Advocacy Careers (AAC) was founded in late 2019 through Charity Entrepreneurship’s incubation program, with the goal to attract top talent to the [effective animal advocacy] movement and empower them to make a real impact.” (Animal Advocacy Careers, n.d.-a)

AAC is a typical example of a capacity-building organisation, aiming to reduce the unnecessary suffering and improve the welfare of animals by supporting those who themselves work on this issue directly.

In this report we aim to assess AAC’s success and effectiveness in achieving this goal. To start with, in this section, we will first examine whether AAC is addressing an important problem within the cause area of animal welfare (1.1), and then describe AAC’s different programmes (1.2). This builds the foundation for the following sections, where we then evaluate AAC’s work.

1.1   Needs assessment

On their “About” page, AAC writes:

“The animal advocacy movement is desperately in need of passionate, talented professionals to help fast forward the end of animal suffering. Recruiting the perfect fit for a role can be an arduous process, with organisations often spending months searching for the ideal candidate for some roles.” (Animal Advocacy Careers, n.d.-a)

AAC also writes that talent bottlenecks can have many detrimental effects on the movement. Among others, they can slow down the growth of organisations or lead to less effective work due to lower quality hires (Animal Advocacy Careers, n.d.-a). In this section, we aim to evaluate these claims, understanding the problem that AAC is trying to solve and how important it is.

Charity Entrepreneurship incubation

The logical starting point for our assessment is the research done by Charity Entrepreneurship (CE) which led to the creation of AAC. In the publication of their top charity ideas for 2019, CE explains that finding the right staff is a frequent challenge for animal advocacy organisations (Charity Entrepreneurship, 2019). In a survey of leaders and researchers within the movement, they found some excitement around the possibility of increasing the talent pool in the animal advocacy movement, especially in large neglected countries like China or India. CE estimated this to be a tricky problem to work on, but one that could be of very high impact. (Savoie, 2018) Due to the large uncertainties involved and the lack of systematic research that has been done on this issue, CE suggested that ways of improving hiring and training within the movement should be investigated and tested further. This research was supposed to be done by a new organisation focused specifically on animal advocacy careers. Apart from research, this new organisation could act as a central contact point for talent and career topics. (Charity Entrepreneurship, 2019)

CE recommended that the organisation would conduct small experiments in three different areas, trying to understand what is the most effective approach for filling talent gaps:

1. Finding (e.g., finding new applicants through headhunting or outreach),
2. Training (e.g., training possible candidates in management skills that are highly lacking in the movement), and
3. Sorting (e.g., matching people with fitting roles). (Charity Entrepreneurship, 2019)

Animal Advocacy Careers’ own research

Following these recommendations and the incubation by CE, AAC started conducting research into different potential priority areas and published the results from a variety of research projects on the blog on their website in late 2020. These findings were supposed to help them in their strategic prioritisation and the design of their services, but could also be useful to others. The following is an overview of different topics they researched:

We do not describe the findings of these research projects here, as they do not seem relevant for our needs assessment.

However, one research project conducted by AAC is highly relevant here. In 2020, AAC published the results of surveys they had done with HR experts and directors of nine animal advocacy organisations, trying to understand their biggest talent and hiring bottlenecks and how they could be addressed (Animal Advocacy Careers, 2020f). Crucially, when it comes to the importance of an influx of stronger talent into the movement, AAC reports the following finding:

“All organisations unanimously agreed that if their quality-adjusted pool of applicants doubled for the next 3 years (vs staying constant), this would contribute more to enable them to do good as an organisation than a doubling of their funding over the same period. The majority agreed they would be able to double the amount of good but some more cautiously estimated 25% more good.” (Animal Advocacy Careers, 2020f)

Following these initial, brief surveys, AAC ran an enhanced version of this research, conducting two surveys with different sets of participants (Animal Advocacy Careers, 2021):

1. Leadership and hiring managers at effective animal advocacy nonprofits:
   AAC reached out to 56 organisations, achieving a 70% response rate with 39 distinct organisations participating. The survey was designed to gather a single representative response from each organisation, specifically targeting individuals in top-level leadership positions such as CEO, ED, COO, VP, or those involved in hiring or human resources. The majority of respondents, with the exception of three, held leadership roles within their respective organisations. Further information on the kinds of organisations that took part in the surveys (e.g., their geographic location, their size, or the focus of their work) can be found in AAC’s report.
2. Researchers, grant-makers, plus others working on “meta” and movement-building services for the movement:
   For this survey, AAC adopted a less formal approach in terms of inclusion or exclusion criteria, reaching out to organisations involved in research, grant-making, and other movement-building services, where they anticipated valuable input. They did not imply a need for the responses to be representative of the entire organisation, but in most instances, they still requested only one response per organisation to minimise the time required for completing the survey. They contacted 21 organisations and received 16 responses from 13 different organisations, resulting in a 62% response rate.

Below, we summarise the findings from these surveys in depth, as they offer crucial insights into the talent needs of the movement and form the basis for AAC’s strategic prioritisation.

AAC’s surveys confirmed that the availability of high quality applicants for paid roles was a highly important bottleneck, and that this finding was consistent across regions. Lack of funding was the key bottleneck in both surveys, with lack of (qualified and capable) applicants for paid roles not far behind. Other highly important bottlenecks were unfavourable attitudes from targets of advocacy and a lack of public awareness of the organisations’ work. The ranking of items from most to least limiting was similar for both the Global North and South. The lack of qualified and capable applicants was particularly highly rated as a bottleneck in Asia and for corporate/producer welfare campaigns. (Animal Advocacy Careers, 2021)

In line with their previous survey, AAC also identified that a high threshold is required for individuals to help animals more by choosing careers focused on donating money to nonprofits rather than dedicating time and labour directly, even for roles which are not necessarily the hardest to hire for. They note that one “could interpret these results as suggesting that organisations would rather receive one additional very-high quality applicant for the roles that are hardest to hire for than receive $25,000 or more (each year for the length of time that the applicant might otherwise have been employed for)” (Animal Advocacy Careers, 2021). However, this finding should not be taken extremely strongly due to concerns about reasoning and interpretation of the corresponding survey question. The finding is also somewhat at odds with AAC’s other finding that funding is the biggest bottleneck for the organisations in their survey.

Nevertheless, the report provides moderately strong evidence that the availability of high quality applicants for paid roles is a major concern for animal advocacy organisations.

More specifically, organisations that participated in the survey faced the most substantial hiring difficulties for roles focusing on leadership or senior management, fundraising or development, and government, policy, lobbying, or legal tasks. However, on average, this difficulty peaked at 3.6, where 3 represented somewhat difficult and 5 represented very much. There were generally higher difficulties in meeting talent bottlenecks in the Global South and Asia, compared to the Global North and not Asia. There were few differences by role type in terms of retaining staff, though fundraising and development stood out as slightly more difficult than average. Retention challenges were considerably lower for each position than hiring challenges, suggesting that responders found hiring a larger issue than retaining talent. To address existing talent bottlenecks, respondents commonly suggested headhunting and training for current staff as solutions. (Animal Advocacy Careers, 2021)

Continuing their efforts of identifying the crucial talent bottlenecks in the movement, AAC published further research into the hiring challenges faced by organisations in 2022. This most recent survey was less extensive than the previous one explained above, as AAC did not expect things to have changed drastically over a relatively short period of time and aimed to reduce the effort for participating organisations. (Animal Advocacy Careers, 2022) The key findings are thus only summarised shortly here as follows:

“As with some of our previous evidence, this survey suggests that senior leadership roles are most difficult to hire for, followed by fundraising and development roles. There were some results that differed from our previous research, such as an absence of evidence that “IT or software” roles are particularly difficult to hire for.

We also asked some questions about what might make candidates for roles seem more or less promising. One surprising finding here was that having an additional year of volunteering with effective animal advocacy nonprofits using relevant professional expertise was evaluated more favourably than having an additional year of somewhat relevant paid experience.” (Animal Advocacy Careers, 2022)

Overall, the evidence from AAC’s own research suggests that talent is a major bottleneck for animal advocacy organisations, specifically in fundraising, leadership, management, and development. While funding was identified as the largest bottleneck for organisations, increased fundraising talent may go some way towards addressing this.

Further external evidence

Apart from the research conducted by AAC themselves and their ‘parent organisation’ CE, we also have to investigate external evidence to understand whether the problem that AAC is trying to address is really as important as they claim it is.

On their website, Animal Advocacy Careers (n.d.-a) make the case that capacity building, the area of work which AAC’s services belong to, is relatively neglected in the effective animal advocacy movement. First, they quote a study from Animal Charity Evaluators (ACE) from 2018 to highlight that capacity building receives only an estimated 2.5% of all resources spent by the U.S. farmed animal movement (Animal Charity Evaluators, 2018). Second, they link to an EA Forum post by their co-founder Jamie Harris (2019), highlighting that there are strong movement building initiatives in Effective Altruism generally, but that these tend to neglect animal welfare as a cause area and focus instead on more long-termist objectives. This provides some evidence that addressing talent bottlenecks might be a neglected issue, just as other capacity and movement building activities.

Rethink Priorities seems to corroborate the view that talent constraints are a high priority issue in their 2023 survey of 35 key decision-makers in the animal advocacy community. They find that the most lacking are experts on the developing world or specific neglected but populous countries, fundraisers, and government and policy and/or lobbying experts. However, as shown in the graph below, they also report that the majority of respondents does not find it either ‘quite’ or ‘somewhat’ hard to find outstanding candidates who want to work at their organisation. (Rethink Priorities, 2023)

The share of respondents who had indicated that it is hard to find outstanding candidates was higher in the previous year’s survey (Rethink Priorities, 2022). This provides some evidence that talent gaps might be declining. This could be a result of AAC’s work on filling talent gaps, but it is hard to establish a causal effect here.

Overall, the contrary evidence cited is not strong enough for us to suggest that talent gaps are not still a major bottleneck for the animal advocacy movement. The bulk of the evidence suggests that AAC’s focus area of improving the talent base within the movement is an important problem to work on. However, Rethink Priorities’ findings highlight the need for continuously monitoring this issue, understanding whether AAC might hit diminishing returns and the movement might experience a talent overhang (compared to other scarce resources) at some time in the future. There is a possibility that AAC’s services will be less needed in the future.

This view is supported by the fact that a variety of capacity-building initiatives have emerged within the movement over the last years. Organisations like the Center for Effective Vegan Advocacy, ProVeg, Scarlet Spark or The Mission Motor run courses, trainings, accelerators, and incubators, or provide mentoring and consulting. Other organisations focus on building the capacity of the movement in certain geographic areas, like Animal Advocacy Africa or Animal Alliance Asia. Still others specialise in careers and talent bottlenecks in the movement-adjacent alternative protein space, like the Good Food Institute (e.g., talent database) and Tälist. This emergence of broader capacity-building services needs to be monitored in order for AAC to coordinate with others and direct its resources towards the most important bottlenecks and most cost-effective solutions. We are not concerned about this though. It is our understanding that AAC is actively coordinating with other actors like the ones mentioned above to make sure that there is no duplication of efforts and the most important needs of the movement are addressed. The fact that AAC played an important role in the initiation of two of the organisations mentioned above strongly supports this view.

Summary

Overall, we find that talent indeed seems to be a major bottleneck in the movement. While funding seems to be an even greater need for the movement, organisations are still willing to trade off significant amounts of money for access to better candidates. AAC is thus addressing an important problem. However, it is important to monitor this need over the next few years, as talent gaps are gradually being filled and other capacity-building initiatives emerge.

1.2 Programme description

AAC works to empower animal advocates and organisations who themselves work on improving welfare directly. Their primary pathway to achieving this is through strengthening the talent pool available to animal advocacy organisations. Apart from that, as secondary streams towards impact, AAC also supports organisations in their hiring practices and aims to strengthen the wider ecosystem of farmed animal advocacy (i.e., by increasing donations to the movement and supporting the creation of new organisations).

In order to achieve their goal, AAC is running a variety of programmes. In this section, we provide an overview of these programmes, using publicly available information from AAC’s website and internal descriptions that AAC has made available to us. We therefore do not provide sources below, except for specific quotes or examples. We only include programmes that are either still active, have been active until recently, or still have relevance for AAC’s current activities. For instance, AAC’s management and leadership training is not included here, as it was suspended years ago and is not relevant anymore.

Career advice resources

AAC is providing career advice to individuals that are interested in contributing to the movement by publishing freely available content on their website. They showcase a wide variety of resources that can help people create a strong career in animal advocacy. On the one hand, they write and publish their own reports and posts. This includes, among others, their skill profiles (here), an “Essential Guide for Animal Advocacy” (here), a series of animal advocate profiles (see for example here), a series of posts about career paths (e.g., here and here), or explorations of certain specific ideas (see for example here). On the other hand, they compile other/external helpful and relevant resources here.

Earning to give

A fairly recent addition is AAC’s programme on earning to give. Given that there is a limited number of roles available in the movement and not everyone will be a good fit for those or be willing to completely change career paths, AAC has decided to more actively encourage people to take the Giving What We Can pledge or otherwise make significant donations to effective animal advocacy organisations. This programme aims to contribute to a more diversified and robust funding ecosystem for the movement. In order to fulfil this aim, AAC develops website content and marketing materials to spread their message, hoping for this to be a potentially low-cost and high-return way of bringing value to the movement. As such, this programme is highly related to the career advice resources mentioned above and we do not include it separately in our theory of change in section 2. 

Career advice calls

AAC is also giving career advice in direct calls with people, supporting them in their career planning and concrete next steps. While the first stream of career advice above provides generic information and input, this second stream is more personalised and targeted. This service is aimed at only very promising individuals due to the larger time investment needed to conduct these calls. They have trialled both 1-1 and group formats over the last years, focusing more on the latter lately in order to improve the cost-effectiveness of this service.

Online course

AAC is running a free, self-paced “Introduction to Animal Advocacy” online course. The course is based on the research and resources described above in the first stream of AAC’s career advice activities and provides a relatively basic introduction to a potentially high-impact career within the farmed animal advocacy movement. With an expected time commitment of roughly 9-18 hours participants are supposed to discover relevant career options, design their career plans, and define actionable next steps. Participants do not have to do the full course, but can pick and choose sessions. Successful participants receive a certificate and are invited to be included in AAC’s talent database which is shared with hiring managers (see below). This is a scalable programme that requires much less staff time per person than personalised career advice.

Job board

AAC hosts a job board on its website, highlighting roles at impact-focused animal nonprofits and key details such as location and application deadline. The board is updated weekly based on AAC’s active research and suggestions by others. Because of AAC’s primary aim of connecting individuals with roles directly within the animal advocacy organisations, the board does not include potentially high-impact roles in academia, government, food companies, or similar. For such roles, AAC refers to other resources (e.g. skill profiles, other job boards). Related to this job board is an AAC mailing list that informs interested people directly about current relevant opportunities.

Fundraising work placement

AAC has run a fundraising work placement programme for the first time in 2022. The programme was informed by previous AAC research on the needs and bottlenecks in the movement (Animal Advocacy Careers, 2022) and aimed to place promising individuals into remote, paid, temporary fundraising positions at effective animal advocacy nonprofits. Participating individuals first did independent learning with AAC’s support for 10-30 hours and were connected with other fundraisers and advocates, before being matched with a partner organisation. They were compensated for the full process. They then moved on to work full- or part-time at the partner organisation for three months, aiming to fill common fundraising bottlenecks and thereby increase the organisation’s impact. Participants were supposed to develop important skills, make connections, and build experience, potentially getting hired by their organisation for a job after the temporary placement.

Recruitment

AAC has experimented with different direct recruitment services. At the beginning, AAC directly supported organisations with recruitment for leadership, management, and fundraising roles, as these were harder to fill than others and could significantly affect the organisations’ impact. In addition, AAC occasionally offered guidance on organisations’ hiring and recruitment practices, either as a substitute for or in addition to their primary search efforts. However, this form of recruitment service was discontinued based on internal evaluation results (see section 3.1.2.2). It is therefore also not included in the theory of change we outline in section 2.

Top candidate database

Instead, AAC decided to trial a new version of recruitment service from October 2022 onwards: their top candidate database. This database includes candidates for which participating organisations report that they have been unsuccessful in an application with them but seem promising. The database is privately shared with partner organisations to consider in their hiring processes. The aim is to avoid the drain of talent from the movement and increase access to strong candidates for animal advocacy organisations.

Inclusive hiring skills workshops

In collaboration with Tania Luna, AAC has run a series of three workshops for hiring professionals (e.g., HR managers, founders) of roughly 90 minutes each. Originally, tickets had to be bought for attending these workshops (at USD 249 for all three workshops as a bundle), but the sessions are now freely available on AAC’s website based on email signup. The content focuses on strategies for successful talent acquisition and diversity, equity, and inclusion (DEI) practices, aiming to build better teams at animal advocacy organisations. Participants are supplied with tools, skills, insights, and a community of like minded people. The idea is nicely summarised by this quote from AAC’s website: “Let’s come together to make the world a better place for animals by making the workplace better for people” (Animal Advocacy Careers, n.d.-c). This programme is different from AAC’s other work, as it focuses on hiring practices instead of the enhanced access to strong candidates. It therefore also follows a different path to impact in the theory of change that we outline in section 2.

Summary

In sum, we have identified the following services that seem relevant for AAC’s theory of change: career advice calls, career advice resources, fundraising work placement, inclusive hiring skills workshops, job board, online course, and top candidate database. Here we exclude the earning to give and recruitment programmes described above, since the former is subsumed under the career advice resources and the latter was suspended in 2022.

2 Theory of change

2.1   Causal chain

In order to understand exactly how AAC aims to tackle the need described in section 1.1 with the programmes described in section 1.2, we have constructed a theory of change (ToC) for their work. The ToC outlines the different steps that need to hold true in order for AAC to have the desired positive effect on animal welfare.

In constructing this ToC, we first aimed to take an unbiased outside view, trying to understand the different pillars of AAC’s work and how they fit together to create positive change. We actively refrained from looking at existing ToCs about AAC’s work at this first stage. Once we felt comfortable about our ToC, we consulted existing ToCs (a simple one from AAC’s website, another very simple one from Animal Charity Evaluators, and a more comprehensive internal one provided to us privately by AAC) in order to see whether we had missed any crucial considerations or had made any mistakes. We also consulted AAC staff for feedback on some open questions. We found that the ToC we had constructed was reasonable and did not include major mistakes. However, we detected some minor omissions that we then added to our conceptualisation (e.g. the potential for AAC to lead people to impactful paths outside of the narrow animal advocacy movement that can help improve animal welfare, like earning to give or working on alternative proteins).

The full visualisation that resulted from this process is shown in the picture below and is available with better visibility here. The outputs are the seven programmes identified in section 1.2. The numbers indicate assumptions that need to hold for the different steps to work out as intended. Their colours represent the level of uncertainty for this specific assumption/part of the ToC prior to a deeper evidence review, from red (high), to yellow (medium), and green (low).

2.2   Outlining assumptions and levels of uncertainty

In this section we outline the assumptions that we identified in the ToC shown above, highlighting some preliminary thoughts and considerations, before we try to resolve as many uncertainties as possible via our evidence review in section 3.

1. AAC can find out which are the most important bottlenecks and most promising careers in animal advocacy.

This assumption seems extremely important for AAC’s ToC. If they cannot find out what the most important bottlenecks and most promising careers in animal advocacy are, they cannot give helpful advice. Their impact would then mostly be in helping people to find opportunities/roles (see assumption #6), giving animal advocacy organisations access to a candidate database (see assumption #8), and improving the hiring skills of these organisations (see assumption #9). Our initial assumption is that AAC’s biggest impact comes from skill-building and providing research and resources to interested individuals (assumption #13). We therefore think that AAC would be left with only a fraction of its impact, if assumption #1 broke down and all of the tightly related services (career advice, online course, and fundraising work placement) could not positively impact people’s career trajectory.

Overall, it seems plausible that AAC can arrive at findings that marginally improve career decisions. Our needs assessment in section 1.1 has already described some research that seems relevant in this regard. It is important for AAC to continuously monitor this though, which we understand they do.

2. AAC is able to publish research and resources that are interesting/relevant to a wide-enough target audience.

It seems plausible that content might not be used/consumed by many people. Only if the resources are used can they have an impact on people’s careers.

However, the assumption is not absolutely necessary to hold for AAC to have the ultimate impact they aim for, since there are other programmes / parallel paths. If we find that the resources are not widely used, AAC might consider shutting down this part of their activities. This seems unlikely to be a good course of action though, since the underlying research and resources are important for other parts of their works as well (direct career advice, etc.). Even if this public part of AAC’s research would not provide much direct external value, the research should generate a solid foundation for their other services.

3. AAC is able to give direct career advice that is interesting/relevant to a wide-enough target audience.

If the direct career advice service is not frequently requested/used, this part of AAC’s services could be suspended, as it is not crucial for other parts of the ToC to hold.

Our prior familiarity with AAC’s work suggests that this is an attractive service for people and is well-received.

4. AAC is able to run an online course that is interesting/relevant to a wide-enough target audience.

Our initial perspective on this assumption is the same as for assumption #3.

5. AAC is able to run a fundraising work placement programme that is interesting/relevant to a wide-enough target audience.

Our initial perspective on this assumption is similar as for assumptions #3 and 4.

However, this programme was only run in 2022 and seems not to have been repeated so far due to funding constraints (Charity Entrepreneurship, n.d.). This might imply that funders did not judge this programme as a good use of resources. Whether this is true or whether there are other reasons for the programme having only been run once is an important aspect to check in our review.

6. AAC is able to curate an overview of opportunities/roles that is interesting/relevant to a wide-enough target audience.

This is an important assumption that needs to hold in order for AAC to be able to help individuals find high-impact roles and opportunities. In theory, AAC could also suspend this service and only focus on other services. As described under assumption #1, we assume that AAC’s largest impact comes from equipping people with the advice and skills they need to have a strong career in animal advocacy (assumption #13). This could also be done without facilitating the search for roles/opportunities. In practice however, we think that there is a strong interplay between equipping people with advice and skills on the one hand and showcasing potential roles and opportunities on the other hand. It is thus important for this assumption to hold in order for AAC to achieve its full impact on talent availability in the animal advocacy movement.

It seems fairly straightforward to curate an overview of opportunities and roles. Our prior familiarity with AAC’s work suggests that this is an attractive service for people and is well-received.

7. AAC is able to do effective outreach and build a strong network of individuals and organisations that want to use their services.

Strong marketing, outreach, and networking is an important supporting factor for all of AAC’s services. On the one hand, it helps AAC to drive more individuals towards their services, thereby increasing the value of their services to the movement. On the other hand, a strong network of partner organisations also helps to enhance AAC’s offering and make it more attractive to individuals. This assumption is thus an important one for AAC to have a strong impact.

Our prior familiarity with AAC’s work suggests that they have marketed their services well and dispose of a good network of individuals and organisations.

8. AAC is able to maintain a candidate database that is interesting/relevant to animal advocacy organisations.

Our initial perspective on this assumption is similar as for assumptions #3 and 4.

If the candidate database is not frequently used, this part of AAC’s services could be suspended, as it is not crucial for other parts of the ToC to hold. We do not know how well-received the service is though.

9. AAC is able to run inclusive hiring skills workshops that are interesting/relevant to animal advocacy professionals.

This part of the ToC seems fairly detached from AAC’s other streams of impact. It should constitute a very minor part of their overall impact. As such, this assumption is not necessary to hold. We still decided to include this aspect of AAC’s work in order to get a comprehensive picture of all the ways in which they aim to improve the lives of animals.

If the workshops are not frequently requested/used, this part of AAC’s services could be suspended, as it is not crucial for other parts of the ToC to hold. We do not know how well-received the workshops are. The fact that they are now offered for free, while they previously had a significant price tag, suggests that demand was not too high.

10. Consumers of AAC’s services decide to found their own organisations.

AAC’s focus seems to be on getting people into roles directly at animal advocacy organisations. However, some people might be influenced by AAC’s career advice to start their own organisation in animal advocacy. Given the large potential value of charity entrepreneurship, the value of this could be significant, even if it only affects a small number of people. We are unsure whether AAC’s work has had any significant effects in this respect so far.

11. Consumers of AAC’s services enhance their career plans and capabilities for work outside of direct animal advocacy that still helps the animal advocacy movement or to improve animal welfare.

Similar to assumption #10, AAC might also influence people to change their career paths into adjacent areas that help animal welfare, like working on alternative proteins or earning to give to animal causes. This should probably not be the main path of AAC’s impact on people’s careers, but it could have significant value and should be considered. In fact, this path might become more relevant now and in the future, as AAC has started a dedicated earning to give programme, as described in section 1.2.

12. Consumers of AAC’s services increase their donations to or otherwise cause counterfactual funding increases for effective animal advocacy organisations.

The earning to give path directly influences people’s donations to effective animal advocacy organisations. However, AAC’s services might also have an effect on the donations of people that are not explicitly choosing a career path focused on earning to give. We are unsure whether AAC’s work has had any significant effects in this respect so far.

In addition, AAC’s fundraising work placement programme directly aimed at strengthening the funding situation for participating organisations.

13. Consumers of AAC’s services enhance their career plans and capabilities for work in direct animal advocacy.

This assumption is crucial to hold for AAC to have its desired ultimate impact, as a lot of their services are geared towards exactly this outcome.

If the programmes are well-designed and well-received, this assumption should be likely to hold as well. We have to look for evidence on how AAC influences people’s career plans and capabilities. In the optimal case, we can break this down by service, to see how much impact each of the different services has.

We also have to consider counterfactuals here, namely what other paths the consumers of AAC’s services would have pursued in the absence of AAC’s services. Overall, the validity of this assumption is crucial for AAC’s overall impact.

14. Consumers of AAC’s services increase their awareness of relevant roles and opportunities.

As written before (see assumption #6), we assume that AAC’s largest impact comes from equipping people with the advice and skills they need to have a strong career in animal advocacy (assumption #13). This could also be done without facilitating the search for roles/opportunities. In practice however, we think that there is a strong interplay between equipping people with advice and skills on the one hand and showcasing potential roles and opportunities on the other hand. It is thus important for this assumption to hold in order for AAC to achieve its full impact on talent availability in the animal advocacy movement.

If the preceding assumptions hold (AAC’s programmes are well-designed and well-received and they are able to curate and distribute relevant opportunities), this assumption should be very likely to hold as well.

15. AAC’s activities cause an influx of more and/or better talent to animal advocacy organisations.

This assumption is crucial to hold, as it arguably describes the key outcome of AAC’s work (based on our understanding and statements made by AAC)..

If the preceding assumptions hold , this assumption should be likely to hold as well. However, since this assumption is so crucial, we have to look for evidence on how AAC’s work influences organisations.

16. AAC’s workshops improve the hiring processes of animal advocacy organisations.

As written before (see assumption #9), this part of the ToC seems fairly detached from AAC’s other streams of impact. It should constitute a very minor part of their overall impact. As such, this assumption is not necessary to hold. We still decided to include this aspect of AAC’s work in order to get a comprehensive picture of all the ways in which they aim to improve the lives of animals.

The impact of workshops on organisations seems uncertain. If workshops do not have an impact on hiring processes, this part of AAC’s services could be suspended, as it is not crucial for other parts of the ToC to hold. We need to evaluate this.

17. Better candidates and hiring processes lead to better staff.

This assumption seems very straightforward and we will not review it in detail here.

18. Better staff allows organisations to do more effective work for animals.

This assumption seems very straightforward.

However, we need to consider here that a lack of talent may not be a main limiting factor for organisations to do more effective work. Instead, they may lack funding or may be inhibited by other factors. Our needs assessment in section 1.1 indeed showed evidence that a lack of funding, unfavourable attitudes from targets, and a lack of public awareness all ranked before a lack of talent when it comes to the most important bottlenecks. However, lack of talent still emerged as a major limiting factor and there is some evidence that organisations prefer talent over donations in certain cases. Overall, this assumption should hold.

19. Increased funding allows organisations to do more effective work for animals.

This assumption seems very straightforward, as our needs assessment in section 1.1 has shown that funding seems to be the major bottleneck for the movement. We will therefore not review this assumption in detail.

20. Career paths outside of existing animal advocacy organisations enhance the movement’s ability to help animals.

This assumption seems very straightforward. If individuals choose to found their own animal advocacy organisation or decide to work in adjacent sectors (i.e. policy, alternative proteins), this should positively influence the movement’s ability to effectively help animals.

21. More effective work for animals improves animal welfare.

This assumption seems true by definition (almost tautological) and we will not review it in detail here.

3    Evidence review

In this section, we use evidence to inform our view of the validity of the different steps and assumptions in the ToC described in section 2. We first look at evidence collected by AAC themselves in section 3.1, aiming to give an overview of AAC’s monitoring and evaluation (M&E) efforts, explain what we can learn from them, and make recommendations about improvements. We then additionally consult external evidence in section 3.2, in order to get a broader understanding of AAC’s streams of work. In section 3.3, we summarise how this analysis of internal and external data updates our view on AAC’s ToC and the key steps and assumptions it involves. Readers looking for a compact summary of our findings are invited to jump to section 3.3 directly.

3.1   Evidence collected by the organisation

Our review of the evidence collected by AAC mainly distinguishes between two kinds of evidence: monitoring and evaluation evidence. These different aspects are often also called ‘process evaluation’ and ‘impact evaluation’ (BetterEvaluation, n.d.). The former aims to understand whether the programmes are properly implemented and how they could be improved (e.g., by understanding completion rates and satisfaction scores of a course), while the latter is focused on the observed changes caused by the programmes, also taking counterfactuals into account (e.g., improvements in participants’ capabilities or changes in their career outcomes that are a result of participation in a course). We consider as monitoring evidence everything that covers activities and outputs in our ToC (assumptions #1 to 9). The remaining part of the ToC (assumptions #10 to 21) is analysed with the help of evaluation evidence, as it is concerned with the outcomes and impact of AAC’s work.

We primarily use data that AAC has provided to us directly for this evaluation, as this should be the most up-to-date and accurate data available. We only referred to their yearly reviews (for 2020, 2021, and 2022) and impact page as well as Charity Entrepreneurship’s overview page on AAC where this seemed helpful or necessary.

In section 3.1.1, we first give an overview of the kinds of data that AAC is collecting, detailing their M&E approach and its strengths and weaknesses. In section 3.1.2, we then move on to describe the results and findings from this internal data. Across the two sections, we do not only describe AAC’s M&E efforts but also provide our perspective and make recommendations about improvements.

3.1.1 What type of evidence are they collecting?

3.1.1.1 Monitoring evidence

The approach

AAC collects different monitoring data for the different programmes they are running. AAC’s leadership reported to us that this data was scattered until recently, but they now have a comprehensive overview of monitoring data for their different programmes. We used this internal overview to understand what kind of monitoring data is collected for each of the programmes.

Below, we provide an overview of the metrics used for the different programmes described in section 1.2 (in alphabetical order). This list is probably not exhaustive but should cover the most important aspects. We exclude all impact-focused metrics (e.g. number of placements caused) here, since they are covered under evaluation evidence in the next section.

In addition to these programme-specific metrics, AAC also collects monitoring data on their marketing efforts (assumption #7). This was reported to us by AAC’s leadership, but we did not see a comprehensive overview of this data and did not have the time to inquire further on this.

Our perspective and recommendations

Overall, AAC’s monitoring approach seems reasonable and it is positive to see that AAC has an overview of monitoring metrics for almost all of its programmes. These metrics can help to reduce our uncertainties on assumptions #2 to 9.

However, we see some areas that could be reviewed and potentially improved upon:

• We are unsure how the monitoring data is used to make decisions internally. We generally recommend that AAC make sure that all of this data is collected with a specific purpose in mind and that they focus on those metrics that are relevant for their decision making.
• Monitoring data is missing for those services that AAC provides to a broad audience via its website or general online presence, namely the generic career advice resources and its earning to give outreach. While we understand that it is hard to track the impact of activities whose effects are so scattered as for these generic services, we still think that there is some monitoring data that could be collected. For instance, just as for its job board, AAC could track the number of page views for its career advice resources. Even though such data might not provide a strong basis for evaluating the effectiveness of these programmes, they can be a first step towards understanding the relevance of different career advice pages (over time) and improving the services provided on that basis.
• For some of the programmes, AAC only collects ‘hard’ data such as the number of people and organisations affected, but neglects to collect ‘softer’ data that can yield more qualitative insights for improving the services. For instance, both the career advice calls and the online course do not seem to solicit feedback from participants on the services provided, understanding how valuable they are and how they could be improved. This might be a missed opportunity for AAC. However, we acknowledge the limitations of such data (e.g., positivity bias in responses) and understand the associated costs (e.g., effort for both AAC and its advisees). This should therefore be seen as a weaker suggestion than the other ones above.

3.1.1.2 Evaluation evidence

The approach

AAC adopts a meticulous approach to evaluating the impact of its programmes, mainly focusing on an innovative metric: the number of Importance- and Counterfactual-Adjusted Placements (ICAPs). This rigorous method underpins their commitment to steering more talent towards the animal advocacy movement, directly addressing assumption #15 in our ToC: the influx of more and/or better talent into animal advocacy organisations. Moreover, the evaluation of ICAPs extends to encompass people founding their own organisations or working in adjacent sectors like alternative proteins, thereby also informing assumption #20.

ICAPs are calculated via a three-step process:

1. Identifying potential placements:
   AAC's first step aims to detect new placements for people who previously participated in AAC’s programmes. This involves the quarterly monitoring of LinkedIn profiles and conducting annual surveys, as well as direct feedback from organisations.
2. Interviewing placements:
   Once potential placements are identified, AAC tries to conduct interviews with each individual to assess the counterfactual impact of their services. This critical step determines the extent to which AAC's intervention contributed to the placement's success.
3. Assessing and adjusting placements:
   The final step involves adjusting the placement score based on the counterfactuals identified in the interviews and making a subjective importance assessment based on the organisation's and the role's significance. These adjustments transform the placement figures into ICAPs. For each ICAP, AAC estimates an addition of $37,500 to the movement. This calculation is based on the assumption that each placement is at least 10% better than the next best candidate and will continue working in the movement for 7.5 years, with an average salary of $50,000 annually. In section 4, we will provide more details on the reasoning behind these calculations and describe our perspective on their validity.

Next to their ICAPs metric, AAC also monitors significant donations influenced by their efforts. This tracking is facilitated through communications with Giving What We Can, where pledgers are asked about contributing factors to their decisions, and AAC's annual surveys. For their most recent earning to give programme, AAC is collecting the number of giving pledges they caused. AAC has also tracked the funds raised and time saved for organisations participating in the Fundraising Work Placement programme. This involved surveying and interviewing participant organisations and assessing counterfactuals to determine the extent of the funds raised due to the programme. For the Fundraising Work Placement programme, AAC also collected further evaluation evidence, specifically how many participants changed their career plans after the programme. 

Based on all of this data, twice a year, AAC conducts evaluations of each programme's performance to decide whether to stop, pivot, continue, or scale them up. This decision-making process is particularly challenging for newer programmes with fewer ICAPs. In such cases, AAC sets specific goals and failure points that programmes must meet to continue. Early-stage evaluations rely on programme data, feedback from organisations and candidates, and potential assessments relative to other AAC programmes. For more established programmes, the ICAPs generated are compared against the effort invested to estimate cost-effectiveness (see section 4). Some write ups of detailed evaluations of specific programmes are publicly available here and here.

Our perspective and recommendations

Overall, the description above documents an elaborate and rigorous evaluation approach. The use of ICAPs is an especially innovative and strong method to measure outcomes.

However, every approach will inevitably have its limitations. Below we list the most important limitations for AAC’s current methodology. We also explain if and how these limitations should be addressed, considering the costs associated with potential enhancements. 

• Assumptions and omissions in ICAPs calculations:
  • AAC’s approach for identifying placements possibly misses placements not represented on LinkedIn or through survey responses. This issue is especially relevant in developing countries. However, the current process employed by AAC seems reasonable and should cover the majority of placements influenced. Getting a completely exhaustive overview of all placements does not seem realistic and would be associated with excessive effort.
  • AAC anticipates a time lag of 6-24 months between intervention and measurable results, meaning their most recent impact estimates (2023) could be significant understatements. This is an inherent limitation given the longer time periods over which AAC’s impact unfolds. Noting this limitation and considering it in evaluations in decisions seems sufficient. 
  • AAC experiences some drop-off in interview responses in step two and estimates that about 80% of their impact is traceable through these interviews. Once again, we think that this is a natural limitation for interviews and think that AAC’s current response rate is acceptable.
  • All of the three bullet points above suggest that AAC’s ICAP estimates are conservative. AAC could add a factor in their calculations that accounts for such missing data. However, we appreciate that AAC only considers the impact of those placements that they know they have influenced and does not artificially inflate their ICAP estimates. We agree that it is better to err on the conservative side here.
  • AAC’s assessment of counterfactuals and the importance of certain placements is subjective. They try to guard against this issue by averaging the scores given by two people. To ensure the accuracy and impartiality of these judgments, AAC has also sought feedback from external stakeholders and experts. We have also done some sanity-checks on their scores ourselves (see section 4). We think that AAC could improve its process here and reduce noise in its judgements, by implementing clearer guardrails on how to score different placements (e.g., via a scoring rubric or framework).
  • AAC’s estimates for the monetary value of an ICAP are based on crucial assumptions that reasonable people could disagree about. We provide more details and our perspective on this in section 4.
• Volunteering roles/placements are not tracked and not included in the ICAP calculation. We are unsure why AAC decided to drop this measurement, but suspect this to be due to the low expected impact and high data collection effort involved. The fact that AAC is not tracking this implies that some positive impact of their work might not be tracked and makes their ICAP estimates more conservative. We are unsure whether AAC should track such volunteer placements as well but suspect that AAC has good reasons not to do so.
• AAC is not tracking attitude changes, career plan changes, or other behavioural changes (e.g., diet change). This seems reasonable, as AAC should focus on the most important metrics in order to avoid interview/survey fatigue.
• AAC does not seem to robustly evaluate the impact that its services have on hiring practices, specifically when it comes to the inclusive hiring skills workshops. AAC had initially planned to evaluate these workshops in more depth via follow-up surveys. However, they decided to suspend this service, as another organisation was founded that would focus specifically on this issue (Scarlet Spark). Given these developments, a thorough evaluation is not planned anymore. We think that this decision is reasonable, also given that this service was quite separate from AAC’s other programmes.
• Just as for the monitoring evidence (see section 3.1.1.1), AAC is missing a method to gather evaluation evidence for their career advice resources and earning to give programmes. While we understand that it is hard to track the impact of activities whose effects are so scattered as for these generic services, we think that AAC should attempt to capture the effects of this as well. In our interview, AAC’s leadership indicated that this issue is on their agenda.
• AAC does not measure how much faster individuals entered the movement due to their interventions, like other organisations such as 80,000 Hours do. This seems fine to us, as their impact on people’s careers is thoroughly assessed via their ICAPs metric.

3.1.2  What are the results from that monitoring and evaluation evidence?

As we now understand how AAC collects its internal data, we can describe what the results from that data collection look like in practice. As in section 3.1.1, we start with monitoring data (section 3.1.2.1) and then move on to evaluation data (section 3.1.2.2).

3.1.2.1 Monitoring evidence

The hard data

It is hard to give an overview of monitoring data that allows for cross-programme comparisons. As described in section 3.1.1.1, AAC collects different monitoring evidence for the different programmes it is running. In addition, the programmes have been run for different time frames and with different levels of effort invested. We therefore only record the numbers in the table below for readers to get a rough sense for different metrics across the different programmes for certain time frames. The numbers should be taken with caution, as we are unsure whether we read AAC’s data correctly. They should be directionally correct though. The programmes for career advice resources, earning to give, and recruitment are not included, based on the analysis in section 3.1.1.1.

As described in section 3.1.1.1, we did not have access to a comprehensive overview of AAC’s monitoring data on marketing activities and did not have the time to inquire further on this.

Further context and our perspective

In addition to this rough overview, we can provide some further insights and our perspective on the different programmes:

• Career advice calls have been a steady component of AAC’s work since the very beginning. The number of calls fluctuates over the different years. We expect this to be the result of this service being very time-intensive and AAC experimenting with 1:1 versus group calls. We know that the demand for these calls is high, but AAC needs to prioritise and can only allocate certain amounts of time to this service.
• The initial fundraising work placement programme seems to have been well-received. The list of partner organisations for this initial programme seems impressive, including Animal Ethics, Sentient Media, Veganuary, Animal Advocacy Africa, and many more. The number and quality of applicants was also strong. In addition, AAC has conducted an assessment of the diversity of applicants and candidates for this programme. We did not check this in detail due to the limited amount of time available for this report.
• The data for inclusive hiring skills workshops is very scarce and unclear. This makes sense in light of AAC’s decision to drop this service, as described in section 3.1.1.2. We therefore did not investigate this further.
• The job board is steadily growing, both in terms of the number of roles listed and organisations included, as well as the number of people visiting and using it. This is a positive indication that the programme is running well.
• The online course has also been a steady component of AAC’s work since the beginning. The number of people who sign up, enrol, and complete the course has grown steadily over the years. We observe a slight drop in completion rates over time, but nothing that seems worrisome.
• The top candidate database has been steadily growing for the first few months of operation. It seems too early to draw any conclusions here though.

These findings positively update us on assumptions #3 (career advice calls), 4 (online course), 5 (fundraising work placement), and 6 (job board). A slight positive update on assumption #8 (top candidate database) could also be warranted. The data does not help us to reduce our uncertainty around assumption #2 (career advice resources), 7 (marketing and outreach activities), and 9 (inclusive hiring skills workshops) though.

3.1.2.2 Evaluation evidence

The hard data

When it comes to evaluation evidence, the crucial metric AAC uses are ICAPs. As described in section 3.1.1.2, ICAPs are calculated for all programmes except for career advice resources, earning to give, and the inclusive hiring skills workshops. Since AAC has a comprehensive overview of ICAPs generated by each programme, we can directly replicate their evidence in the table below. It represents all ICAPs generated by AAC in the years 2019 to 2023 YTD. The last column sets the number of ICAPs generated in relation to the amount of resources invested into the programme. This allows us to draw conclusions on the relative effectiveness of the different programmes.

Further context and our perspective

These numbers are highly relevant, but of course need to be put in context. Below, we provide some further insights and our perspective on the different programmes:

• Due to their central position in AAC’s evaluation efforts, it is important to check the validity of the ICAP estimates above. In order to do so, we perform some sanity checks and sensitivity analyses in section 4. For now, we will take these values as given.
• The career advice course, job board, and online course are the three most cost-effective programmes for AAC. These three programmes are well-established and AAC should continue to run them in the foreseeable future.
• Career advice calls seem cost-effective, but require large time investments. AAC is thus trying out different ways to make this even more efficient.
• The online course is also well established and cost-effective, but is much more scalable.
• The job board seems to be the most cost-effective programme when it comes to creating ICAPs. In addition, since it is fairly difficult to collect feedback from job board users (as for the other website content, as highlighted earlier), the number of ICAPs caused by AAC might be significantly higher and the cost per ICAP even lower.
• The cost per ICAP is relatively high for the fundraising work placement, mostly due to the high effort involved and the large funding amounts needed. However, a large share of participants has landed a permanent role after the programme. Next to ICAPs, AAC has measured the amount raised by participants for the partner orgs, estimating roughly USD 270,000 added and raised counterfactually. Overall, this seems like a high-quality but also high-effort service. AAC seems somewhat positive about the programme and is aiming to run this again or facilitate a similar programme run by another organisation. The focus of this next iteration would be on legal and policy roles instead of fundraising. We are unsure whether this should be a priority area for AAC, as its other services seem more effective and the costs seem too high (see section 4.1).
• As described in section 1.2, AAC has suspended the recruitment programme. The low number of ICAPs generated was a main reason for this. The top candidate database was set up to succeed this service. This new programme is still in its early stages and ICAP numbers should not be relied on heavily at this point yet. AAC seems to be happy with the initial KPIs they are seeing and will need to monitor how the ICAP metric develops over time.
• We did not have the time to analyse in detail how AAC’s services influence people’s donations.
• Due to a lack of data, we could not understand the outcomes and impact of AAC’s career advice resources, earning to give programme, and the inclusive hiring skills workshops.

Overall, the evaluation evidence demonstrates that AAC has counterfactually caused people to change career paths and assume roles within the movement. This positively updates us on assumptions #10 (founding of new organisations), 11 (strong career paths in related areas), 13 (strong career paths and capabilities for work in the movement), 14 (increased awareness of opportunities), and 15 (influx of talent to the movement). The evidence on assumption #12 (increased donations and funding) is less strong, given that we have seen evidence of increased funding as a result of the fundraising programme but have not had the time to analyse the effect of AAC’s services on people’s donations and the earning to give programme is still in its early stages. We could also not update our uncertainty around assumption #16 (improved hiring practices at organisations) due to a lack of evaluation data for the workshops. Assumptions #17 to 21 were not investigated in detail due to our low uncertainty around them and the absence of relevant data.

3.2  External evidence not collected by organisation

Apart from the evidence that AAC has collected internally, we also need to look at external pieces of evidence that can inform our opinion about the kinds of programmes AAC is running and how likely their ToC is to hold. We specifically took the following steps in order to broaden our perspective on AAC’s work and reduce key uncertainties.

1. We did a quick, time-capped literature review on assumption #13 in our ToC, as it is one of the most crucial breaking points in our ToC and we thought we could reduce our uncertainty on this assumption with the help of external evidence
2. We reviewed evaluations of EA career interventions that are similar to AAC’s work, looking for findings that might be cross-applicable. Most importantly but not exclusively, we looked at the findings from 80,000 Hours’ annual reviews since 2019.
3. We consulted Animal Charity Evaluators’ review of AAC in 2022, especially looking for crucial aspects that we had not considered and checking how their perspective lines up with ours.

3.2.1  Academic studies on the effect of career services on career plans and capabilities

Due to the limited time available for this research, we decided to focus our efforts in looking for external academic evidence on the plausibility of AAC’s ToC on assumption #13: “Consumers of AAC’s services enhance their career plans and capabilities for work in direct animal advocacy”. Among the three most crucial and uncertain assumptions in the ToC (#1, 13, and 15), this was the assumption we thought could clearly benefit the most from external research. Assumption #1 was already covered by other pieces of evidence (as described in our needs assessment in section 1.1) and #15 seemed highly specific to AAC’s work and a review of external literature did not seem likely to yield highly relevant insights.

As a result, we focused our attention on the question how likely different AAC services are to influence career plans and capabilities of advisees. In line with the ToC, we focused our attention on those services that aimed to influence these outcomes, namely AAC’s career advice (both in generic as well as personal form), online course, and fundraising placement programme. We investigated each kind of service in a quick literature review, using standard tools like Google Scholar or Elicit.

Career advice

In order to evaluate the effects of career advice, we grouped the generic online-content version and the more personal and direct version of AAC’s services together, since some of the best studies we found included different kinds of career advice interventions.

For instance, one random-effects meta-analytical study from 2023 sought to determine the “impacts of career guidance interventions on school students' career-related skills, knowledge and beliefs” (Nargiza et al., 2023), evaluating empirical studies carried out over the previous 10 years, including nine studies involving a total of 1,433 participants. The analysis resulted in a weighted mean effect size of 0.42 (95% confidence interval = 0.19, 0.65; z = 3.61, p < 0.01), which they interpreted as a moderate-to-high effect size, noting a significant difference between the treatment and control conditions at post-treatment. They found that post-test career-related outcomes in students which received career guidance were significantly higher than for students who did not receive career guidance. This suggests that career guidance interventions may provide some modest developmental progression in school-age children and adolescents, “particularly through improving learners' career decidedness and attitudes such as future time perspective” (Nargiza et al., 2023). Notably, the types of interventions varied between studies, but included short to medium term interventions such as a training programme to improve emotional intelligence, a programme aimed to promote learners’ future time perspective and to learn about the current world of work and vocational planning. (Nargiza et al., 2023)

Similarly, another meta analysis of different career choice interventions found that the targets of such interventions scored roughly one third of a standard deviation higher on various outcome measures like career maturity, career decidedness, career decision-making self-efficacy, perceived environmental support, and perceived career barriers (Whiston et al., 2017).

A review by Mackay et al. (2015) found that no single intervention or group of interventions appeared most effective in improving career management skills. However, they identified the five components below which underpinned the interventions that substantially increased effectiveness and can be considered by AAC in their intervention design:

• The use of narrative/writing
• The importance of providing a ‘safe’ environment
• The quality of the adviser-client relationship
• The need for flexibility in approach; the provision of specialist information and support
• Clarity on the purpose and aims of action planning

Specifically related to direct, personal career advice, we found some further evidence. Due to a lack of abundant meta analyses or systematic reviews, we decided to include one well-designed non-meta study of career counselling. This study found decreases in career indecision and increases in life satisfaction for recipients of career counselling. “The majority of clients implemented their career choice within a period of one year; some partially implemented it; others changed their career choice, rather successfully; and few people did not demonstrate advancement in either their choice or its implementation during this period of time” (Perdrix, 2012). We also decided to include a meta analysis of the effects of mentoring here, since mentoring and career advice calls share significant similarities. This meta analysis on the effects of mentoring finds benefits on career outcomes at small effect sizes (Allen et al., 2004).

Overall, we find strong evidence that career advice can boost capabilities, career outcomes and plans.

Online Courses

Allen et al. (2006) conducted a meta analysis to compare the test performance of students that engaged in distance versus traditional learning. They found that “distance education course students slightly outperformed traditional students on exams and course grades”. The U.S. Department of Education (2010) reported similar findings in their meta analysis, showing modestly stronger performance of students in online learning versus face-to-face settings. Mathieson (2010) systematically reviewed the results for statistics courses specifically and found student performance to be similar between online and face-to-face learning. Gao et al. (2021) found that exam performance for medical courses in China was higher in participants of massive open online courses versus traditional courses. In sum, all of the relevant meta analyses and systematic reviews we identified in our quick review of the evidence point at the ability of online courses to positively influence student’s capabilities.

When it comes to the effect of online courses on career plans, we could not identify any strong studies in the short time available. The best evidence for this clearly seems to be AAC’s (2023) own evaluation of the effects of their career advice and online course on career-related outcome measures. This was already summarised in section 3.1.2.2 and we can neither positively or negatively update our view based on external evidence here.

Overall, we find good evidence that online courses are an effective tool for learning and can improve participants’ capabilities. The effect on career plans is not clear.

Fundraising placement programme

Inceoglu et al. (2021) conducted a systematic review of quantitative studies that either offered a control group or a longitudinal design, seeking to evaluate the effectiveness of placements for career outcomes. They screened 2394 studies and retained 40 for an in-depth review. They identified that “Placement participation elicits an overall positive (but small) effect on career outcomes”. Most notably, graduates who completed placements found employment more quickly, and students’ perception of their own self-efficacy, knowledge, skills, and attitudes were improved. 

A report by the UK Department of Education (2021) corroborates the general finding that work placements have a small, positive association with career outcomes. Looking at employability programmes and work placements in UK Higher Education, they find that “research looking at students’ own assessments tended to find almost ubiquitous evidence of perceived skill development”. Indeed, they conclude that even when statistical techniques are employed to account for selection effects from ability or motivation, “a positive, albeit possibly smaller, wage return to work placements is seen [...], as well as a positive impact on employment outcomes such as the likelihood of being invited to interview”. However they also note several concerns with the methodology and sparsity of evidence available in this field. In particular, they identify that “even in the literature on opportunities, costs and benefits for students or graduates, the research evidence tended to be piecemeal rather than comprehensive.” Further, they note that skill development does not always translate into improved labour outcomes, and posit several explanations for this, such as that not all work placements develop skills equally and that work placements are associated with improved career outcomes as they serve as signals to employers rather than necessarily increasing skills.

Another systematic review, focused on international internships and skill development, found that in 9 out of 49 studies reviewed, career decision-management was identified as a skill gained through participation in international internships by students in higher education (Di Pietro, 2022).

Gillespie et al. (2020) also found positive relationships between internship participation and career development measures (including professional development, career adaptability, career commitment, and career exploration) and learning outcome measures (such as GPA and skill development) in their review. They concluded that the positive effects of internship participation seem greater from voluntary than for compulsory internships and that the positive effects diminish over time. Further, they corroborate the suggestion of the aforementioned studies, that the role of internships is as a signal that students acquired greater skills, knowledge, and productivity.

Overall, we found reasonably strong evidence of a small to moderate positive association between internship or work placement completion and career outcomes, measured through a range of factors including skill development, self-perception, employment outcomes, and career adaptability. However, there is also a suggestion that this association is explained best by the signalling hypothesis, that internships serve as a signal of skills to employers, and that this may not necessarily mean capabilities themselves are significantly improved. Despite this uncertainty, it seems plausible that such placements improve capabilities and career outcomes. It should be noted that the majority of studies concern students in higher education, which AAC’s programme was not limited to. Further, skill development may be higher in effectively designed fundraising placements which AAC’s programme provides than at the baseline for such placements. Nevertheless, this provides enough collective evidence that placements tend to have positive effects upon capabilities and career plans.

Overall, the evidence for the effect of placements and internships on capabilities and career plans seems even stronger than for the other services. This is in line with the finding in section 3.1.2.2 that the work placement programme is a high-quality and high-effort service.

Summary

In sum, we find good evidence for assumption #13 of the ToC to hold. If properly implemented, AAC’s career advice, online course, and fundraising programme should improve participants’ career plans and outcomes as well as capabilities. This matches with the internal evaluation evidence collected by AAC described in section 3.1.2.2.

3.2.2  Evidence from similar EA programmes

On their “About” page, AAC cite 80,000 Hours (80K) as a successful example of an organisation that provides career advice and addresses talent bottlenecks in the adjacent effective altruism community. They suggest that similar services focused specifically on animal issues could have similarly positive effects. (Animal Advocacy Careers, 2019) We agree that evidence from highly related programmes is an important form of evidence for us to consider here. While it might be less rigorous than the academic studies cited above, this evidence is more directly applicable to the case of AAC and can thus give more practical insights.

80,000 Hours

When looking at programmes that are similar to AAC’s and how they can inform our perspective on AAC’s work, 80,000 Hours indeed stands out.

As 80K started already in 2011 (80,000 Hours, n.d.), they have been able to review and refine their services over a longer time span than AAC. For instance, they already did surveys in 2014 that seem highly insightful into the kind of impact that career advice services can have. In these surveys, they found that 63 out of 206 respondents said that engagement with 80K had changed their career plans. Other sources apart from their surveys informed them of another forty people that had changed plans because of them. These career changes were based on multiple different 80K services like 1-1 advice and events, which all seemed impactful. Roughly one third of plan changes were the result of just reading 80K’s online content, showing that online content alone can be impactful, without deeper engagement levels (like direct, personal advice). In addition to plan changes, 80K also found that their services successfully introduced people to the EA community and changed people’s attitudes towards career. (80,000 Hours, 2014)

Later reviews by 80K found that their initial impact estimates were probably overly optimistic and had to be corrected downwards. In 2019, they wrote the following:

“In past years, we were overly credulous when the people who made plan changes claimed we permanently changed the trajectory of their careers and our investigations into these claims were insufficiently skeptical, thoughtful and thorough. This led us to overestimate our impact, especially on those people who already had some ties to the EA community, because there are already other influences pushing in the same direction. The team has become substantially less likely to take claims about our impact at face value, which we think is reflected in the new system we’re developing for evaluating our impact.” (80,000 Hours, 2019)

80K’s work generally stresses that tracking the impact on career plans is very hard, especially because such changes occur over long periods of time. To account for this, 80K uses qualitative insights as an additional perspective in their evaluations. (80,000 Hours, 2019)

80K’s reviews in later years still showed a positive impact of their work. However, there are some negative aspects or doubts that we need to highlight here. First, it seems that 80K’s work has hit diminishing returns, as their services have grown over the years (80,000 Hours, 2020). Second, 80K found its job board to be less cost-effective than other services and considered cutting the service back or spinning it off. They decided to continue running it, as it seems like a low-effort resource that the EA community should have at its disposal. However, they also decided against investing further into this service and kept it running at low capacity. (80,000 Hours, 2019, 2020) Third, 80K decided to pause its headhunting service in 2022, focusing more on career advice. The reasons for this seem to have been mostly administrative (i.e. not finding a good person to lead this aspect of their work) and not related to the service’s effectiveness. Fourth, 80K found that their content made some of their readers feel sad/stressed/demoralised, as they came away feeling that they did not have what it takes to make a meaningful contribution (80,000 Hours, 2020). Fifth, the collapse of FTX and the case of Sam Bankman-Fried made 80K aware of other potentially negative effects of their work, causing them to update some of their advice. Their decision to focus more on doing research and improving their advice in 2023 also seems tightly connected to these developments. (80,000 Hours, 2022).

These experiences seem highly relevant for our assessment of AAC’s programmes and their (potential) impact. 80K generally offers quite similar services, just at a larger scale and with a broader focus, and can thus act as a good comparison. Below we describe the key takeaways from our review of 80K and how they update our perspective:

• Overall, 80K’s success in a highly similar field updates us positively on the likelihood of impact and effectiveness of AAC’s programmes. Particularly, the general online career resources and the direct, personal advice should be effective in changing people’s career plans. 80K’s experience with the job board seems less promising, but it still seems like a service worth offering. This leads us to positively update our perspective on assumptions #2 and 3, and to a lesser extent #10, 11, 13, and 14.
• 80K’s initial overestimation of their impact highlights why counterfactuals need to be taken seriously when evaluating impact. As described in section 3.1.1.2, we think that AAC has properly incorporated counterfactual considerations into their approach, adjusting the placements they influenced for importance and counterfactuals. While counterfactuals are hard to account for, AAC’s approach seems sufficiently rigorous. The fact that 80K was and is also struggling with their M&E efforts and find it hard to assess their own impact shows how difficult such an exercise is in the realm of meta interventions like AAC’s.
• 80K’s decision to pause its headhunting services seems different from that of AAC to suspend their recruitment service. While 80K’s reasons were mostly operational/administrative, AAC had concerns about the effectiveness of the programme. It still seems notable that both organisations experienced difficulties with this approach, making us sceptical of the value of recruitment services. This affects assumption #15, as recruitment services are aimed at directly increasing the availability of good talent in the movement.
• AAC needs to watch out for diminishing returns in the services they provide. This seems like a natural development in many fields and should keep AAC vigilant in their assessment of their own programmes and in making strategic decisions.
• There are significant potential negative effects to giving wrong, misleading, or otherwise harmful career advice. On the one hand, AAC needs to mind the potential of their content to alienate its consumers. On the other hand, they should stress to their advisees’ the importance of wider, long-term consequences of their actions, especially when it comes to outsized risk taking and unethical behaviour. These concerns caution us in our assessment and should be actively considered by AAC. They slightly negatively update our view on assumption #1, as well as on assumptions #15 and 20, as it seems more uncertain that AAC can find the most promising careers in animal advocacy and give correct advice that will lead to positive changes in people’s career paths.

Centre for Effective Altruism

Apart from 80,000 Hours, there are some other programmes within EA focused on careers and movement building that bear some similarity to some of AAC’s work. For instance, the Centre for Effective Altruism is trying to grow the effective altruism movement by running fellowships and virtual programmes, supporting university groups, organising conferences and retreats, and other services. We checked some of their reporting to see whether it can yield relevant insights for our evaluation of AAC.

However, most of the information we found did not prove to be too helpful. While they have a public data dashboard and regularly posts updates about their work on their blog (Centre for Effective Altruism 2021, 2023), almost all of the data the Centre for Effective Altruism reports on seems to focus on activities and outputs. For example, they list the number of attendees of retreats and EA Global conferences, the number of university groups supported via advice and grants, and the number of participants in fellowships and virtual programmes. They do report on some more advanced measures like people’s likelihood to recommend retreats and conferences, as well as completion rates for their fellowships and virtual programmes. Philip Hall Andersen’s (2022) review of introductory EA courses also mostly focuses on monitoring and process evaluation, but does not look at the further effects on career choices.

We did find one EA Forum post (2019) on the Centre for Effective Altruism’s work that contained some useful information. Evaluating 1-1 meetings they organised for participants in EA virtual programmes, they found that such meetings generally seem valuable, but less scalable and more time-consuming than other services. Especially valuable seem those 1-1s focused on careers and networking, with mentors that are experienced in a specific cause area (e.g. animal welfare), sociable, and very knowledgeable. Mid-career professionals are mentioned as the most promising target group for such 1-1s. The organisers think that the meetings can be impactful when it comes to influencing people’s career paths. All of these findings are tentative and do not seem based on an extremely rigorous evaluation. They should thus be taken with caution, but can provide a weak form of evidence for our review.

Below we summarise our key takeaways from the review of the Centre of Effective Altruism and how they update our perspective

• Most of the data we found is not comprehensive enough and too dependent on the specific implementation of CEA’s programmes such that it cannot be used to inform our view of AAC’s work.
• The findings from evaluating 1-1s seem applicable to AAC’s work and match some of the findings from their evaluation. Specifically, they positively update us on assumption #3 (the ability to run a well-received direct career advice programme) and, to a lesser extent, #10, 11, and 13 (changes in career plans based on AAC’s activities), showing that personal meetings to discuss careers can be a valuable service to the community.

Animal-focused organisations

Further relevant external evidence could come from other capacity-building organisations in the animal advocacy movement. Especially career-oriented services like the talent database offered by the Good Food Institute or Tälist’s services could be investigated. Due to the limited time available for this report, we decided not to pursue these paths here though. Also, Tälist might be too young (they started in 2022) to already have a good understanding of the impact of their work.

Summary

In sum, we find highly relevant evidence from similar programmes within the EA movement. The overwhelming majority of evidence updates us positively on a variety of assumptions in the ToC. However, we also found some reasons for caution that need to be strongly considered by AAC.

3.2.3  2022 Animal Charity Evaluators review

In 2022, Animal Charity Evaluators (ACE) evaluated AAC in order to determine whether they should be recommended to donors as one of their top or standout charities. ACE’s review was overall very positive but did not end up recommending AAC because other charities scored higher on the kinds of programmes they pursue and their cost-effectiveness. (Animal Charity Evaluators, 2022)

ACE thinks that AAC’s strategic priorities are important ones, targeting high-priority countries and focusing on high-priority outcomes and interventions. Capacity building is seen as a neglected but important area and the impact of research is considered promising, especially in the long term. The online course, fundraising placement programme, and job board were found to be the most cost-effective of AAC’s services, while the recruitment service scored lowest. All programmes scored moderate to high in terms of cost-effectiveness. It is important to note that the review did not seem to include AAC’s written career advice, mailing list, and inclusive hiring workshops. (Animal Charity Evaluators, 2022)

Overall, these findings provide the following updates to our view of AAC’s work:

• The review by ACE does not provide much significant new information that we had not already gathered in previous sections of this evidence review. The overall picture of AAC’s programmes and organisation is very positive, which matches our perspective thus far.
• The recruitment services offered by AAC are an issue to monitor, as they seemed least cost effective to ACE and were already restructured by AAC (as described in section 3.1). Investigating the effects of AAC’s top candidate database (the new implementation of recruitment services) should be an evaluation priority going forward.

3.3  What have we learned and not learned?

The evidence we have collected in this section gives us a much better understanding of AAC’s work and impact. In the table below, we summarise the assumptions from the ToC described in section 2 and how we update our views on each of them based on this evidence review. The colours in the ‘Assumption’ column match those given in section 2.2, indicating our levels of uncertainty around the assumptions, while the colours in the ‘Update’ column represent how the different factors we identified update our view. We naturally weigh strong, internal, or case-specific evidence more heavily than weak, external, or general evidence. The different shades of red and green in the ‘Update’ column account for this. We then assign a final uncertainty score from 1 (low uncertainty) to 5 (high uncertainty).

This overview illustrates that the evidence review positively updates us on most assumptions in the ToC and that we generally have few significant uncertainties.

Below we list the most important remaining uncertainties and our perspective on them:

• To what extent can AAC make sure that the services they provide do not have negative externalities by pushing people into paths that are not the best option for them, causing anxiety over career decisions, or causing overall harm to the movement?
  • This seems hard to guarantee. The best way to avoid negative externalities is probably to do rigorous research into the best career paths and carefully consider all options. Specifically monitoring potential negative effects could also be helpful.
• Should AAC continue to pursue recruitment services in some form or another, given that previous experiences with these services do not seem too promising?
  • AAC’s leadership has shared with us that there is a strong demand for these services from the organisations in their network. Recruitment services also seem highly demanded in the for-profit sector, so they should be helpful in the nonprofit sector as well. It might just be about finding out which kind of service works best. We remain somewhat sceptical about this and think this needs to be monitored closely, as it might simply be that there is no recruitment service that can be as cost effective as AAC’s other services.

We will summarise these and all of our other recommendations for AAC in section 7. Before we do that however, we first need to have a more detailed look at the cost-effectiveness of AAC’s programmes in the next section, the health of the organisation as a whole in section 5, as well as their future plans and room for more funding in section 6.

4   Cost-effectiveness analysis

In this section we dive deeper into the cost effectiveness of AAC’s services. To do so, we build on the evaluation evidence described in section 3.1.2.2, namely AAC’s ICAP estimates per programme. In section 4.1, we first replicate AAC’s own cost-effectiveness estimates by comparing the effects described in section 3.1.2.2 with the total costs. We then move on to evaluate AAC’s methodology in arriving at these cost-effectiveness estimates in section 4.2.

4.1 Summary of AAC’s internal cost effectiveness estimates

The following table summarises AAC’s expenditures per programme and overall, since conception. More than 50% of expenditure came from the ‘Other’ category, which we cannot easily associate with a specific programme. The most costly individual programme was the fundraising work placement, which is very noticeable, since the programme has only been run in 2022. Career advice calls, the online course, and recruitment all required similar proportions of the total expenditure. The job board takes only a small share of the overall costs. The same is true for the top candidate database, which was only started recently.

Combining the costs with the ICAPs generated per programme, AAC can estimate a USD cost per ICAP, as already shown in section 3.1.2.2. The table below summarises AAC’s effects and total cost effectiveness per programme, and as a total of the organisational activities, from conception. The number of ICAPs generated can be multiplied by the assumed USD 37,500 value of one ICAP, as explained in section 3.1.1.2, in order to arrive at a USD estimate of value added to the movement. 

The relative cost effectiveness of the different programmes was already described in section 3.1.2.2. Combining the total value generated for the movement via ICAPs (USD 1,814,933) with the total costs (USD 704,416), AAC achieves a multiplier of USD 2.6 generated for every USD spent.

4.2 Evaluation of AAC’s methodology

Given the limited time available for this section of the report, we determined that we would be unable to produce a higher quality metric to measure impact than AAC’s own ICAPs. We therefore decided to instead examine AAC’s methodology and calculations in depth and make recommendations for how they could be improved. First, in section 4.2.1, we look at how AAC estimates the ICAPs generated by a given placement and how robust these estimates are. We also make recommendations for how this could be improved. Second, in section 4.2.2, we analyse the calculation underlying the monetary value of one ICAP and make recommendations for how it could be made more robust.

4.2.1  ICAP estimates per placement

Calibration of counterfactual estimates

In order to estimate the ICAPs generated by a certain placement, AAC follows three steps:

1. Identifying the distribution of impact AAC had upon the participant’s placement over each service engaged with by the participant
2. Identifying the importance of a placement to the movement by assigning an estimated multiplier for the importance of their organisation, relative to the average EAA organisation
3. Identifying the counterfactual impact that AAC had on this placement by estimating the probability of the individual not placing a role within the movement within the next five years

In order to test the validity of AAC’s importance judgments (step #2 above), AAC have sought independent judgement calibration from other organisations.

However, the counterfactual judgments (step #3) have not been tested. Therefore, in order to test the validity of these judgments, one of us (Aashish) performed a calibration and sensitivity analysis on the values assigned for the estimated probability of the individual not placing a role within the movement within the next five years.

The methodology we employed was the following:

1. Selected 5 participants at random
2. Outlined a rubric for evaluating the counterfactual impact AAC had on their placement
3. Read the interview transcripts that AAC conducted with them and marked them against this rubric
4. Assigned a score
5. Compared scores to AAC’s score

The results of this exercise can be seen in the table below. For the sake of confidentiality, no identifying information about the participants will be provided.

Notably, there was a low disparity between our scores and AAC’s scores. This provides some evidence for the robustness of AAC’s estimates. However, given our limited knowledge of the animal advocacy job space, we would not over-index on our own estimates of the probability of the individual not placing a role within the movement within the next five years. 

We recommend that AAC solicit feedback from more experienced people to calibrate their judgements. In addition, AAC currently does not have a clear rubric or framework for coming to well calibrated estimates. This may lead to some margin of error or variation in scores. As described above, we created a simple rubric for assigning our scores in this calibration exercise. We recommend that AAC develop something similar for their own use. Below, we list the factors used in our calibration, which can serve as inspiration and starting point for AAC to build upon.

• Motivation
  • Was the participant interested in working in animal advocacy?
• Initiative
  • Have they taken any previous steps to improve their fit for jobs?
  • Have they taken any previous steps to apply for jobs?
  • Did the AAC programme they engaged with increase their motivation to find a job?
• Competence
  • Did they possess a relevant skill set to provide value to an organisation within the movement?
  • Did they possess relevant experience with EAA or the job-type they sought?
  • Did the AAC programme they engaged with increase their skill set?
• Awareness
  • Did they have the relevant network and/or awareness of jobs within the EAA space without AAC?
  • Did the AAC programme they engaged with increase their awareness of opportunities?

Sensitivity analysis on counterfactual estimates

Given that the scores we arrived at differed somewhat from those given by AAC, it is important to determine how significant variation in this metric would be upon the final impact and cost effectiveness estimates. Therefore, we conducted a sensitivity analysis on this metric, adjusting all estimates for counterfactual impact since the beginning of ACE’s impact evaluation.

The table below outlines the sensitivity analysis performed, determining the effect of varying the counterfactual judgement score upon the overall impact and cost effectiveness of AAC’s total activities. For a sensitivity analysis by programme, see appendix A (in the full PDF report). 

This demonstrates that very small variation in the counterfactual judgements that are central to calculating the number of ICAPs generated per placement can result in moderately large changes in the total cost-effectiveness of the organisation. This highlights the importance of systematising the values assigned to this metric.   

As aforementioned, given these results we recommend that AAC take efforts to systematise their counterfactual judgement estimations. The following steps seem promising to us:

1. Use confidence intervals rather than point estimates:
   Currently, two employees of AAC provide point estimates for this metric. To improve robustness and increase reasoning transparency, we suggest that they provide a 90% confidence interval, and take the average of the overlap between these intervals. Our prior is that using ranges rather than point estimates increases the accuracy of the judgement, as one can consider what they believe the upper and lower bound to be, and this also allows for the level of uncertainty and precision to be conveyed. 
2. Develop a rubric:
   A rubric or framework for coming to these judgements is critical, in order to ensure that this process is systematic and controlled.
3. Identify examples of individual cases where the confidence is in the estimate is high for the calibration of scores
4. Determine base rates, upper and lower bounds:
   AAC should determine the extent to which different factors tend to contribute to job placement. Determining the base rate for different broad categories of individuals to land placements in the movement, based on key factors like experience, skill, motivation and awareness, may help to calibrate estimates. Further, determining the expected upper and lower bounds for the probability of any person to land a placements in the EAA movement regardless of the aforementioned factors, would improve judgements at the extremes.

4.2.2  The monetary value of an ICAP

Assuming that ICAP estimates per placement are relatively robust (which they seem to be based on our analysis above and which they might be even more so in the future if our recommendations are implemented), the cost effectiveness estimates of AAC’s work might still be mistaken, if the monetary value assigned to an ICAP is off.

ICAPs are calculated based on a multiplicative model and represent a monetary equivalent of value added to the movement as a result of AAC’s programmes. AAC uses the following formula:

1 ICAP = Average number of years the candidate will continue working in the movement * Average salary of a candidate * Percentage better the candidate is than the next best candidate

For each ICAP, AAC estimates an addition of  USD 37,500 to the movement. This is based on the assumption that each placement is 10% better than the next best candidate and will continue working in the movement for 7.5 years, with an average salary of  USD 50,000 annually. AAC has also added a multiplier for cases where organisations think that the placement was above or below the 10% assumption, to adjust this value.

From our interview with AAC, we understood that the assumption of the number of years a new placement will stay in the movement is a conservative estimate, and is not based on any particular dataset. Given the degree of variation there may be in this value, we performed a sensitivity analysis to determine how changes in this value would impact the total value of one ICAP in the table below. 

The analysis shows that the value generated per ICAP varies drastically, if assumptions about the duration of impact change.

Based on our conversations with AAC, we understand that AAC’s estimate of USD 37,500 per ICAP might err on the conservative side, as other organisations (like 80,000 Hours) use higher monetary values in their evaluations. As such, it seems unlikely that AAC is grossly overestimating its impact.

Nevertheless, it seems highly beneficial for AAC to enhance the foundations that their ICAP estimate is built on. To increase the certainty around the assumption of the number of years an individual stays in the movement, we propose AAC use existing data or collect novel data, to build a probability distribution for this metric that represents the state of the movement. The weighted average from this can be taken, to more robustly determine the average number of years a new placement stays in the movement. This may also be updated periodically, such as every five years or at points where it becomes clear that career norms in the movement have changed and that the previous value no longer holds.

4.3 Recommendations

To summarise, the table below provides an overview of the major recommendations made in this section, including their importance, urgency, and expected time investment.

5 Organisational health and transparency

Apart from AAC’s specific programmes and their effectiveness, we also decided to include a brief analysis of the organisation as a whole. In this section, we will first provide our perspective on AAC’s organisational health (section 5.1) and then on their transparency in communication and data sharing (section 5.2). For this, we will draw mainly on ACE’s 2022 review of AAC and our own experience in evaluating AAC. 

5.1   Organisational health

Overall, AAC’s organisational health seems strong. In their review of AAC, Animal Charity Evaluators (2022) did not detect any reason for concern when it comes to AAC’s leadership and culture. We did not perform a detailed review of this ourselves, but could also not detect any issues during the research for this report.

ACE has one executive staff, Lauren Mee (Chief Executive Offer). Lauren is on the Board of Directors, which consists of three males and three females to provide a gender balanced board. However, Lauren is not a voting member, to avoid conflicts of interest.

According to ACE, all six staff members responded to their survey about organisational health in 2022, and all agreed that AAC’s leadership team guides the organisation competently. Additionally, AAC’s leadership has “two-way feedback sessions” with each staff member to learn about staff morale and work climate.

In addition, ACE examined whether AAC had best-practice general policies in place. We have updated this overview to the best of our knowledge, especially accounting for the fact that AAC has implemented a formal compensational policy since their ACE review.

Apart from these general policies, AAC also has staff policies against harassment and discrimination. We report these policies in the table below, again based on ACE’s review.

According to ACE, no staff report that they have experienced harassment or discrimination at their workplace during the last 12 months before their 2022 review, and no staff report to have witnessed harassment or discrimination of others in that period.

Overall, from consulting ACE’s review of AAC and asking further questions in an interview with AAC leadership, we judge there to be no organisational health concerns. 

5.2   Transparency

In their review of AAC, Animal Charity Evaluators (2022) found the organisation to be moderately transparent to the public and made some smaller suggestions for improving transparency:

“AAC has been transparent with ACE during the evaluation process and during other communications with ACE. A list of key staff members and information about their accomplishments are available on AAC’s website. However, their audited financial documents (including the most recently filed IRS form 990 for U.S. organizations)[...] and a list of board members are not available on their website. Based on the information that they make publicly available, we assess AAC’s transparency towards the public as moderate.” (Animal Charity Evaluators, 2022)

In our own review of AAC, we found them to be very transparent with us, sharing the required data as necessary. The evaluation evidence and financial data was well-structured. They have published annual reviews for each of the last years (2020, 2021, and 2022) and we think that the organisation is generally committed to transparency.

However, we think that some of the information could be more easily accessible to the public. For instance, as far as we could discern, AAC’s detailed methodology for how they calculate ICAPs is not accessible to the broader public. Access to this write-up of the methodology, which is linked to in AAC’s 2021 review and therefore may be intended to be public, is restricted. Their website is also at times hard to navigate, listing some programmes like the fundraising work placements and the inclusive hiring skills workshops in their ‘About’ section. In our conversations with them, AAC indicated that they are aware of some issues with the user-friendliness of their website and are actively working on this.

Overall, we think that AAC’s transparency is high but there is some room for improvement.

6    Future plans and room for funding

This section outlines the AAC’s sources of funding as well as their indicated future plans.

6.1 Where is the funding coming from?

The table below demonstrates the distribution of funding from different donor types. 

The following table below demonstrates the distribution of funding from different foundations. The names of these foundations were anonymised for privacy reasons.

96.3% of AAC’s total funding in 2023 came from seven different foundations. Of this, nearly 90% comes from Foundations A and B which are both EA funding sources. This demonstrates that the majority of funding comes from a small number of larger EA donors.  We expect that money from EA funding sources would have a higher counterfactual value than money from non-EA funding sources. AAC has reported turning down funding from non-EA sources, since these grants were attached to conditions to remove EA content from the website and promote lower impact opportunities.

Based on the analysis in this section, we suggest that AAC should seek to diversify its funding sources as much as possible whilst maintaining the ability to prioritise what they judge to be the most impactful and cost effective programmes and opportunities. This would help both to increase their own financial security and to free up more funds within the EAA space. We estimate a priori that, as a meta-organisation focusing on directing animal advocates to achieve the highest impact possible through their careers, AAC may have more potential to win funding from the wider animal advocacy movement beyond EA funders than direct animal organisations. This is due to their provision of services that can be of value to a broad range of animal organisations and animal advocates that are beyond the effective animal advocacy space, unlike many other EAA charities. Further, given the number of individual users of their services, they may be able to seek small donations from those who have found it beneficial. However, this may require them to somewhat broaden the focus of their programmes, to also include promoting and supporting less effective opportunities and organisations, which may reduce their impact. Therefore, we suggest that AAC should (continue) to seek funding from both EA and non-EA funders where possible, whilst continuing to only accept funding when this does not impede their ability to uphold their focus on promoting the most cost effective opportunities.

6.2 Future plans

AAC seemed to have significant room for more funding beyond AAC’s projected revenues, between USD 200,000 and 300,000 additionally for 2023 and 2024 each. ACE estimated that AAC could productively employ total revenues between USD 800,000 and 850,000 in those two years respectively. (Animal Charity Evaluators, 2022)

For 2024, AAC has set itself the following strategic priorities:

1. Support talented, mission-aligned candidates to maximise the impact of their careers
2. Fully fund the organisation's ambitious budget for 2024.
3. Develop and sustain talent pipelines focused on the key global talent bottlenecks, paving the way for a landscape of international talent density.

AAC has clear measurable and quantified targets for each of these, and evaluation points specified.

7 Summary

This section provides a short summary of the key findings of this review. We decided not to include a summary of all of the recommendations we made throughout this report here, since this would have been too extensive. For more detailed explanations and a wide range of recommendations, see the relevant sections of the report or consult appendix B (in the full PDF report), which includes a copy of all the recommendations made throughout the report.

Overall, our perspective on AAC as a charity is mostly positive. We judge their programmes and M&E systems to be moderately strong, but present a variety of suggestions for how this may be improved. 

We find that AAC addresses an important need, as talent seems to be a major bottleneck in the movement. However, it is important to monitor this need over the next few years, as talent gaps are gradually being filled and other capacity-building initiatives emerge.

The different steps in AAC’s ToC mostly seem well-backed by evidence. We find strong evidence that AAC is pursuing a variety of cost-effective programmes and has counterfactually caused people to change career paths and assume roles within the movement. The main things we are uncertain about when it comes to AAC’s programmes are to what extent AAC can make sure that the services they provide do not have negative externalities and whether AAC should continue to pursue some kind of recruitment service.

AAC's monitoring system covers most programs effectively. However, for broad-audience services like website resources and 'earning to give' outreach, monitoring is somewhat lacking. AAC's use of ICAPs for evaluating its impact is innovative and useful, but there are some limitations that should be addressed.

When it comes to the health of the organisation and its funding situation, we did not find major reasons for concern. We do make minor recommendations for improvements in these areas though, most notably with regards to diversifying AAC’s funding streams.