Thanks for the thought!
You might be interested in the analysis we did in 2020. To pull out the phrase that I think most closely captures what you’re after:
We also attempted an overall estimate. This gave the following picture for 2018–2019 (all figures in estimated DIPY per FTE and not robust):Website (6.5)Podcast (4.1) and advising (3.8)Job board (2.9) and headhunting (2.5)
We also attempted an overall estimate.
This gave the following picture for 2018–2019 (all figures in estimated DIPY per FTE and not robust):
We did a scrappy internal update to our above 2020 analysis, but haven’t prioritised cleaning it up / coming to agreements internally and presenting it externally. (We think that cost effectiveness per FTE has reduced, as we say in the review, but are not sure how much.)
The basic reasoning for that is:
This appendix of the 2022 review might also be worth looking at - it shows FTEs and a sample of lead metrics for each programme.
Cody's answer below and mine above give better 'overall' answers to your question, but - if you'd like to see something concrete and incomplete you could look at this appendix of job board placements we're aware of.
Basically: these just take a really long time!
Lumping 2021 and 2022 progress together into a single public report meant that we saved hundreds of hours of staff time.
A few other things that might be worth mentioning:
Hi Vaidehi - I'm answering here as I was responsible for 80k’s impact evaluation until late last year.
My understanding is that plan changes (previously IASPC's then DIPY's) were a core metric 80K used in previous years to evaluate impact. It seems that there has been a shift to a new metric - CBPC's (see below).
This understanding is a little off. Instead, it’s that in 2019 we decided to switch from IASPCs to DIPYs and CBPCs.
The best place to read about the transition is the mistakes page here, and I think the best places to read detail on how these metrics work is the 2019 review for DIPYs and the 2020 review for CBPCs. (There’s a 2015 blog post on IASPCs.)
Some more general comments on how I think about this:
A natural way to think about 80k’s impact is as a funnel which culminates in a single metric which we can relate to as a for profit does to revenue.
I haven’t been able to create a metric which is overall strong enough to make me want to rely on it like that.
The closest I’ve come is the DIPY, but it’s got major problems:
(There’s a bit more discussion on impact eval complexities in the 2019 annual review.)
So, rather than thinking in terms of a single metric to optimise, when I think about 80k’s impact and strategy I consider several sources of information and attempt to weigh each of them appropriately given their strengths and weaknesses.
The major ones are listed in the full 2022 annual review, which I’ll copy out here:
On your specific questions:
Ah nice, understood!
I don't think you'll find anything from us which is directly focused on most of these questions. (It's also not especially obvious that this is our comparative advantage within the community.)
But we do have some relevant public content. Much of it is in our annual review, including its appendices.
You also might find these results of the OP EA/LT survey interesting.
Hi - thanks for taking the time to think through these, write them out and share them! We really appreciate getting feedback from people who use our services and who have a sense of how others do.
I work on 80k’s internal systems, including our impact evaluation (which seems relevant to your ideas).
I've made sure that the four points will be seen by the relevant people at 80k for each of these.
Re. #1, I'm confused about whether you're more referring to 'message testing' (i.e. what ideas/framings make our ideas appealing to which audiences) or 'long term follow up with users to see how their careers/lives have change'. (I can imagine various combinations of these.)
Could you elaborate?
I was interested in seeing a breakdown of the endpoints, before they'd been compressed into the scales AAC uses above.
Jamie kindly pulled this spreadsheet together for me, which I'm sharing (with permission), as I thought it might be helpful to other readers too.
Through overpopulation and excessive consumption, humanity is depleting its natural resources, polluting its habitat, and causing the extinction of other species. Continuing like this will lead to the collapse of civilisation and likely our own extinction.
This one seems very common to me, and sadly people often feel fatalistic about it.
Two things that feeling might come from:
It's really cool to see these laid out next to another like this! Thanks for posting Katja :)