we agree. Small, cheaper projects that focus on one thing tend to work better. We understand big ambitious projects can have more potential impact and that "impact" is difficult to measure. E.g. if a peace-building project can truly achieve peace, nothing beats that, is just that most peace-building projects are just workshops with people that have no say on whether war or peace will happen.

We prefer avoiding the word impact entirely and talking of effects (intended, visible), this misses some of the biggest opportunities and lots of indirect benefits.

At least a few of the projects could have benefitted from breaking them down into smaller projects and considering them separately in each category. We were going down that road at the beginning but then we thought that wasn't entirely fair to the other participants. Because many organizations were not fully understanding with how they were going to be evaluated, giving one 3 chances was an unfair advantage. 

I think at least one of these projects could have been a finalist on the economic growth section, should it have been broken down in this way, but if the organization applying thought that was a human rights project then we have to compare it on the effects in that category (which we phrased as likelihood of protecting people from abuse, this was the hardest as there is less work on EA on this subject), and then it did not do much but spent lots of money. In any case, this is only relevant for the contest, we tried to have a fair one but that wasn't the ultimate objective. The contest was only relevant to us as a way to identify cost-effective projects that we would like to support further, it was clear this wasn't one of them.

Despite our energetic writing, we may have been carried away. We had very limited tools and information. It would be more accurate to say the first, no evidence of effects.

In fact, we did not have the tools or data to look rigorously into all projects and their intended and unintended effects. 

We had two layers:

In the first layer we assume projects do exactly what organizations claim they do, and just establish a possible output per dollar (well in this case, per franc CFA). If they have usable output or outcome data, we use that, if not we may even use research on an equivalent program (eg. the GBV example). In each category, it is easy to compare which output per dollar is cheaper, still with some working assumptions.

From there, the difference between organizations was quite huge, and we had some budget to do data collection for the possible best 6 (thanks EA Infrastructure fund). There we just try to confirm the effect claimed by interviewing beneficiaries of the assistance. In two cases the effect claimed wasn't visible at the time of data collection and that gave us two finalists (economic, because in the other project most beneficiaries did not remember participating, and health, because in the other project participants appeared to be worse off than before the project), in the human rights category there were two very similar projects as finalists and one had slightly stronger effects. 

We were clearly biased for small budgets, so the overall winner had a big advantage because it was literally an intervention for 1 family, we think this may still be accurate anyway, and it is plausible that there are great opportunities to do good at small scale in developing countries, particularly through cash. 

We had also limitations in comparing across sectors, but more or less the 3 finalists got the same (a badge, a framed award, some feedback they can use with potential donors, and subscription to an online newsletter to funding opportunities. We recognized the human rights final position was more tight and we added the runner-up to the newsletter). We decided to do more for the winner because we thought it was the only one meeting cost-effectiveness expectations and we can't find much better in Cameroon, but that was out of the contest.

Going back to your question, if I have to guess, I am sure these projects may have effects that we did not get to see. I am unsure these effects are achieved in a cost-effective manner because they are buried in so much else.

thank you very much this is appreciated. We did not know we were doing much useful honestly :D, the reaction to this post is very encouraging

There is a level of confidentiality because organizations shared this in good faith and not for public dissemination.

Moreover, any numerical values were very speculative and just for the purpose of comparison. We can only confidently share a broad analysis like the one in this post.

thank you I appreciate this

Hi,

1. There weren't any cash winnings, just prestige (and we fundraised for the best because we liked it enough). We definetly would attract more participants if we had a cash grant at the end. However, the application process did not take time. Unlike most of such processes, we were looking at reports of projects that already took place, and we had no interest and discarded any project proposal for new funding.
2. One of the participants of the fellowship was the director of one of the NGOs that submitted documents. A few participants were at the time or had been members of another local NGO who also participated, but this is a large local NGO and they did not have any role in the application process. Neither of these two organizations was a finalist. 

This is very useful and we could possibly improve the basic model with some of these suggestions

Thank you very much

We are trying to raise funds for families living with children with this condition in Cameroon. We have written about our position in this post Critique our position – Sickle cell anaemia in sub-Saharan Africa (Cameroon) — EA Forum (effectivealtruism.org)

We hope you don't mind if we include our donation link, in case anyone wants to support these children: Help children with sickle cell syndrome - GlobalGiving

We realize your post is about sickle cell as a cause from a Public Health point of view while our approach is more of a Social Protection one.