When it comes to the step between research questions and the probability distribution, full research, answering each question, can be seen in the full report. In the report, we also address some of the concerns you have with the judgement calls on each of the “qualitative” parameters.

Each update incorporates the weight we put on this factor, the directionality and strength. Those factors, again, rely on other information. With the example, you cited ”what ACE thinks makes an effective campaign” vs “probability that all companies defect in a Prisoner's Dilemma scenario". For example, ACE’s opinion on the importance of public support when launching corporate campaigns is formed based on the intervention report they have researched in November 2014, and as they currently claim “is not up to our current standards.”. The landscape has changed since then. As of recent, we can observe that there is a strong track record of successful corporate campaigns in countries where the society didn’t have sympathetic views toward animals (e.g. Lithuania or Japan). I think we can rely more and more on rigorous and generalizable conclusions from research on real-life examples and on the application of game theory to predict the behaviour of the companies.

I agree I wish we had enough time to flesh out the reasoning for each of the factors. Sadly, due to limited time we are constantly having to make trade-offs about whether we should put time into explaining the reasoning more deeply to the broader community vs discussing with the CE candidates vs researching more to get a deeper internal understanding. We generally plan on going deeply into these factors with the specific entrepreneurs looking to start this project or others, who are going to work/are working in the field in the near term, but not publish much more on the topic publicly after our full report.

Another way to interpret the result is that it is your subjective probability that a given company will follow through. But then I’m not sure it makes sense to have a 90% SCI of what your own subjective probability is. What would that even mean? How would you check that the true value is in the confidence interval?

Karolina has already answered, my response here is very late and you've already worked on corporate campaign research, so I'd guess you're already satisfied, but hopefully this will be useful for anyone else who comes by or might still be useful to you.

See, for the frequentist approach, https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval .

Polling, e.g. for elections, is a useful comparison, if the follow-through rate in cage-free campaigns is interpreted as as the proportion of pledged companies that will actually go cage-free, rather than a probability that a given (or random) company will go cage-free.

Here, for each pledged company (ignoring those that haven't pledged), we have a binary random variable (Bernoulli trial) whose outcome is whether or not it follows through on its commitment. In polling, the binary variable for each individual from the population from which respondents are sampled is whether or not they would vote for candidate X if an election were held the day of polling (and then we might extrapolate the estimate to the actual election held in the future, but this would not really be justified under frequentism).

Checking that the true value is in the confidence interval would just mean checking for each pledged company (pledged by a given date, or pledged by the time we check), whether or not it has followed through, and calculating the proportion that have followed through.

In polling, the population of interest is the votes of actual voters on the day of the actual election, but not all poll respondents will vote, and surely not all actual voters were polled. Furthermore, even respondents may change their mind between the day of polling and the actual election, so even if we polled all and only the actual future voters, because they can change their mind, the statistical populations, in the technical sense, between poll responses and actual future votes could still differ, since it's the responses/votes that make up the statistical populations, not the people.

For cage-free campaigns, the population of interest is the set of companies that have made pledges by some date (e.g. before today, so only companies that have already made pledges). If we base our estimate only on companies that have already made pledges, we don't "poll anyone who won't vote", so our sample population is a subset of (possibly equal to) the population of interest. I suppose this might be complicated if we consider companies that have actually followed through by the time we've finished collecting data for our estimate and then start using cages again; this would be like poll respondents changing their minds.

(Or, as done in https://forum.effectivealtruism.org/posts/WvgrGLDBko6rZ5qax/did-corporate-campaigns-in-the-us-have-any-counterfactual or as you've done in https://forum.effectivealtruism.org/posts/L5EZjjXKdNgcm253H/corporate-campaigns-affect-9-to-120-years-of-chicken-life , instead of binary random variables for following through, we could estimate progress by a given date.)

More space per chicken is just one of the requirements. Probably the most important requirement is to use higher welfare breeds, which generally grow more slowly. But there are more requirements regarding lighting, enrichments, etc. You can see the full ask in the European Chicken Commitment. Asks for other regions are similar and can be seen here.