Note: the following is meant to be a short, low-caveat overview for a topic which I’ve previously written perhaps-overly-complex posts about.
Note [5/10/2022]: If you would like to see example applications of this framework or other clarifications/objections, see my previous, long post about this.
Update [3/1/2023]: Some of the language was updated to make it more clear that this framework is about decomposing and evaluating individual "pros" and "cons," rather than providing a framework to weigh competing values (e.g., "Does X amount of privacy outweigh $100M in increased costs?").
Intro and Context: Where Are All the Advanced Pro-Con Analytical Frameworks?
Many people have heard of pro-con analysis: when you are trying to make a decision or conduct policy analysis, you often want to identify and evaluate individual advantages and disadvantages of a given decision.
However, doing this analysis can be difficult, especially if the arguments involve uncertainty or if you already feel biased for or against some of the options.
Thus, I’ve long been confused why there is not also a widely-known or “standard” framework—even among EAs—for doing deeper pro-con analysis. I am especially confused by the non-popularity of such a framework among EAs despite how they already use frameworks/heuristics such as the “importance, tractability, neglectedness” (ITN) framework for broad cause area prioritization (despite its flaws).
The COILS Framework: A Simplified Description
I've long been developing and advocating a framework that I've temporarily[1] labeled the COILS framework (Counterfactuality, Implementation, Linkage, Significance). This framework holds that any given advantage/disadvantage[2] for a decision is built on four categories of claims/assumptions:
Advantage/disadvantage: “This plan would cause X, which is good/bad.”
- Counterfactuality: X would not occur (to a similar extent) without the plan.
- Implementation: The plan details and ultimate implementation will involve doing Y.
- Linkage: X will occur following the plan implementation (Y).
- Significance: X is morally good/bad.
To give an alternate, more-testable statement of this framework: Any rebuttal[3] against any claimed advantage/disadvantage is exclusively one or more of these four categories of claims; these four categories cover all the ways in which a claimed advantage/disadvantage could be wrong. The four concept categories are therefore collectively exhaustive.
At least, I believe the above is probably accurate, although I definitely recognize it is possible I’ve made an oversight for reasons described in this footnote.[4] If you do disagree with such a claim, please let me know why!
Additionally, I also think that the four categories are at least mostly mutually exclusive (i.e., the definitions don't overlap), although I find it is often more efficient in application to not nitpick over the exact delineation between concept pairs like “implementation and linkage” and “linkage and significance.” However, I am less confident about this claim than the claim that they are collectively exhaustive.
Arguments in favor of using/popularizing this framework[5]
I don’t think that using this framework is always optimal, especially for very simple decisions. However, there are two semi-overlapping categories[6] of arguments as to why I think this framework probably ought to be used/popularized among EAs:
Specific-effects justifications:
- It seems helpful for checking some of your key assumptions, especially when you're already biased in favor of believing some argument;
- It standardizes/labelizes certain concepts (which seems helpful for reasons such as easier concept communication among groups and faster concept categorization for one’s own thinking).
Heuristic-level justifications
- Breaking down complex questions/concepts into smaller pieces is generally helpful if the smaller categories are still collectively exhaustive and do not involve significant duplication (i.e., are mostly mutually exclusive).
- The EA community has already deemed the ITN framework to be useful (despite its flaws), and the COILS framework shares similarities with the ITN framework.
- The COILS framework is adapted from a framework popular in competitive policy debate (the “stock issues”)[7], and there are reasons to expect that the policy debate community would have some valid justifications for gravitating around such a framework.
One Potential Objection[8]
One of the biggest arguments in my mind against this framework is the heuristic-level objection: “If this is a good idea that could have been adopted for a while, why hasn’t it been adopted more widely?”
I have a variety of thoughts in response, but to be honest I don’t have a specific, short answer that I feel very confident about, so I’ll just briefly list a few thoughts in this footnote.[9]
Regardless, I would love to hear other people’s feedback/objections!
Conclusion
Ultimately, I think that it could help EA discourse and subsequent decision-making if more people popularized/used this framework, similar to how EAs tend to think the ITN framework has been helpful, but I would love to hear other people’s feedback, including objections. If you would like to read a less-neat and slightly outdated but longer explanation of this framework, including example applications and clearer definitions, you can see this post.
- ^
This is liable to change, especially if I can find better labels for “implementation” (which is actually a category that involves questions related to feasibility and plan details—e.g., the presence/absence of grandfather clauses—in addition to “implementation”) and “linkage” (which is basically trying to say “X outcome occurs in the world with Y implementation” but without claiming “Y causes X,” which is a derived statement from the combination of counterfactuality and “linkage”).
- ^
E.g., “this regulation would save 100 lives by reducing certain pollutants.”
- ^
A caveat/clarification: this does not include redirection responses that do not challenge individual pros/cons, such as “it’s true that your plan would cause X and that is as good as you say it is, but your plan also causes W which is bad enough to outweigh the claimed benefits of X.”
[Update 2/28/23] This also applies to the idea of double-counting between multiple advantages/disadvantages: suppose someone proposing a decision claims an advantage such as "this will save $1B/yr" and then gives a separate advantage saying "this will provide 10K jobs/yr," but this second argument is actually just a subtle extension of the effects of the "$1B/yr saved" argument. In this case, it might be that neither argument is "wrong" individually, but when one tries to do a net-effect analysis of the overall plan—i.e., considering all the advantages and disadvantages—it would presumably be wrong to present these as two separate advantages.
It seems quite plausible that both of these caveats could be addressed through the "significance" portion of COILS (i.e., "you claimed that $1B/yr translates to X utility, but this is incorrect because that does not account for the utility gains via other effects"), but I understand why someone may view this as shaky. Ultimately, I still think the framework is solid insofar as it focuses on individual arguments alone or "the whole plan all at once" (i.e., evaluating all the positive and negative effects of the plan as one large argument), but it is not meant to analyze individual arguments alone in a way that then provides automatic answers to "all of the effects on balance."
- ^
I’ve rarely gotten feedback on the framework from people other than policy debaters, and even then I have rarely ever received criticism—and the criticism I did receive I largely addressed years ago. Thus, most of my refining of the framework has been driven by internal criticism, which I acknowledge might just be missing something.
- ^
Yes, technically I could have analyzed “use this framework” according to the framework itself, but for length reasons and given that I don’t expect the audience to already be familiar with using the framework, I chose not to. Also, “use this framework” is a somewhat ambiguous proposal, which can make formal application of this framework more difficult (although it definitely does not prevent informal/heuristic-level application of this framework, much like how people often heuristically use the ITN framework.)
- ^
I.e., there is the potential for double-counting between the two categories, and even to some extent within the “heuristic-level justifications” category.
- ^
To be clear, stock issues theory is a bit of a mess given how there are different interpretations—partially stemming from the variation in leagues/regions—and debaters will occasionally sacrifice formality and precision for simplicity in rounds, but the overall idea is fairly widespread, especially among my former league.
- ^
I won’t go much into detail on many of the potential arguments against this framework for length purposes and given that I have previously written a short section on that subject. Most of the more-impactful objections to the framework, though, I think would stem from claiming that the framework is inaccurate in some way.
- ^
1. People could have used/formalized the ITN framework for a long time, but I think time has suggested that it is useful in the EA community.
2. Aside from the stock issues in policy debate, this framework probably already does exist but with different names and other partial variations.
3. This could require a critical mass of adoption in a community in order to be worth learning.
I did include some example applications in the long introduction post (see my response to Khorton); I was worried that trying to include an example in this version might make it too long and thus lead to a high bounce rate… but perhaps I should have made it clear that I do have some applications in the old post.
Here is the first example (with updated terminology):