Hide table of contents

Note: the following is meant to be a short, low-caveat overview for a topic which I’ve previously written perhaps-overly-complex posts about.

Note [5/10/2022]: If you would like to see example applications of this framework or other clarifications/objections, see my previous, long post about this.

Update [3/1/2023]: Some of the language was updated to make it more clear that this framework is about decomposing and evaluating individual "pros" and "cons," rather than providing a framework to weigh competing values (e.g., "Does X amount of privacy outweigh $100M in increased costs?").

Intro and Context: Where Are All the Advanced Pro-Con Analytical Frameworks?

Many people have heard of pro-con analysis: when you are trying to make a decision or conduct policy analysis, you often want to identify and evaluate individual advantages and disadvantages of a given decision. 

However, doing this analysis can be difficult, especially if the arguments involve uncertainty or if you already feel biased for or against some of the options.

Thus, I’ve long been confused why there is not also a widely-known or “standard” framework—even among EAs—for doing deeper pro-con analysis. I am especially confused by the non-popularity of such a framework among EAs despite how they already use frameworks/heuristics such as the “importance, tractability, neglectedness” (ITN) framework for broad cause area prioritization (despite its flaws).


The COILS Framework: A Simplified Description

I've long been developing and advocating a framework that I've temporarily[1] labeled the COILS framework (Counterfactuality, Implementation, Linkage, Significance). This framework holds that any given advantage/disadvantage[2] for a decision is built on four categories of claims/assumptions:

Advantage/disadvantage: “This plan would cause X, which is good/bad.”

  1. Counterfactuality: X would not occur (to a similar extent) without the plan.
  2. Implementation: The plan details and ultimate implementation will involve doing Y.
  3. Linkage: X will occur following the plan implementation (Y).
  4. Significance: X is morally good/bad.

To give an alternate, more-testable statement of this framework: Any rebuttal[3] against any claimed advantage/disadvantage is exclusively one or more of these four categories of claims; these four categories cover all the ways in which a claimed advantage/disadvantage could be wrong. The four concept categories are therefore collectively exhaustive.

At least, I believe the above is probably accurate, although I definitely recognize it is possible I’ve made an oversight for reasons described in this footnote.[4] If you do disagree with such a claim, please let me know why!

Additionally, I also think that the four categories are at least mostly mutually exclusive (i.e., the definitions don't overlap), although I find it is often more efficient in application to not nitpick over the exact delineation between concept pairs like “implementation and linkage” and “linkage and significance.” However, I am less confident about this claim than the claim that they are collectively exhaustive.


Arguments in favor of using/popularizing this framework[5]

I don’t think that using this framework is always optimal, especially for very simple decisions. However, there are two semi-overlapping categories[6] of arguments as to why I think this framework probably ought to be used/popularized among EAs:

Specific-effects justifications:

  1. It seems helpful for checking some of your key assumptions, especially when you're already biased in favor of believing some argument;
  2. It standardizes/labelizes certain concepts (which seems helpful for reasons such as easier concept communication among groups and faster concept categorization for one’s own thinking).

Heuristic-level justifications

  1. Breaking down complex questions/concepts into smaller pieces is generally helpful if the smaller categories are still collectively exhaustive and do not involve significant duplication (i.e., are mostly mutually exclusive).
  2. The EA community has already deemed the ITN framework to be useful (despite its flaws), and the COILS framework shares similarities with the ITN framework.
  3. The COILS framework is adapted from a framework popular in competitive policy debate (the “stock issues”)[7], and there are reasons to expect that the policy debate community would have some valid justifications for gravitating around such a framework.


One Potential Objection[8]

One of the biggest arguments in my mind against this framework is the heuristic-level objection: “If this is a good idea that could have been adopted for a while, why hasn’t it been adopted more widely?”

I have a variety of thoughts in response, but to be honest I don’t have a specific, short answer that I feel very confident about, so I’ll just briefly list a few thoughts in this footnote.[9]

Regardless, I would love to hear other people’s feedback/objections!



Ultimately, I think that it could help EA discourse and subsequent decision-making if more people popularized/used this framework, similar to how EAs tend to think the ITN framework has been helpful, but I would love to hear other people’s feedback, including objections. If you would like to read a less-neat and slightly outdated but longer explanation of this framework, including example applications and clearer definitions, you can see this post.

  1. ^

    This is liable to change, especially if I can find better labels for “implementation” (which is actually a category that involves questions related to feasibility and plan details—e.g., the presence/absence of grandfather clauses—in addition to “implementation”) and “linkage” (which is basically trying to say “X outcome occurs in the world with Y implementation” but without claiming “Y causes X,” which is a derived statement from the combination of counterfactuality and “linkage”).

  2. ^

    E.g., “this regulation would save 100 lives by reducing certain pollutants.”

  3. ^

    A caveat/clarification: this does not include redirection responses that do not challenge individual pros/cons, such as “it’s true that your plan would cause X and that is as good as you say it is, but your plan also causes W which is bad enough to outweigh the claimed benefits of X.”
    [Update 2/28/23] This also applies to the idea of double-counting between multiple advantages/disadvantages: suppose someone proposing a decision claims an advantage such as "this will save $1B/yr" and then gives a separate advantage saying "this will provide 10K jobs/yr," but this second argument is actually just a subtle extension of the effects of the "$1B/yr saved" argument. In this case, it might be that neither argument is "wrong" individually, but when one tries to do a net-effect analysis of the overall plan—i.e., considering all the advantages and disadvantages—it would presumably be wrong to present these as two separate advantages. 
    It seems quite plausible that both of these caveats could be addressed through the "significance" portion of COILS (i.e., "you claimed that $1B/yr translates to X utility, but this is incorrect because that does not account for the utility gains via other effects"), but I understand why someone may view this as shaky. Ultimately, I still think the framework is solid insofar as it focuses on individual arguments alone or "the whole plan all at once" (i.e., evaluating all the positive and negative effects of the plan as one large argument), but it is not meant to analyze individual arguments alone in a way that then provides automatic answers to "all of the effects on balance."

  4. ^

    I’ve rarely gotten feedback on the framework from people other than policy debaters, and even then I have rarely ever received criticism—and the criticism I did receive I largely addressed years ago. Thus, most of my refining of the framework has been driven by internal criticism, which I acknowledge might just be missing something.

  5. ^

    Yes, technically I could have analyzed “use this framework” according to the framework itself, but for length reasons and given that I don’t expect the audience to already be familiar with using the framework, I chose not to. Also, “use this framework” is a somewhat ambiguous proposal, which can make formal application of this framework more difficult (although it definitely does not prevent informal/heuristic-level application of this framework, much like how people often heuristically use the ITN framework.)

  6. ^

    I.e., there is the potential for double-counting between the two categories, and even to some extent within the “heuristic-level justifications” category.

  7. ^

    To be clear, stock issues theory is a bit of a mess given how there are different interpretations—partially stemming from the variation in leagues/regions—and debaters will occasionally sacrifice formality and precision for simplicity in rounds, but the overall idea is fairly widespread, especially among my former league.

  8. ^

    I won’t go much into detail on many of the potential arguments against this framework for length purposes and given that I have previously written a short section on that subject. Most of the more-impactful objections to the framework, though, I think would stem from claiming that the framework is inaccurate in some way.

  9. ^

    1. People could have used/formalized the ITN framework for a long time, but I think time has suggested that it is useful in the EA community.

    2. Aside from the stock issues in policy debate, this framework probably already does exist but with different names and other partial variations.

    3. This could require a critical mass of adoption in a community in order to be worth learning.





More posts like this

Sorted by Click to highlight new comments since: Today at 12:12 AM

I think I'd find an example of criticising a decision using this framework interesting

Some example applications can be found in this section of my previous (long) post on this topic, but I will make a slight—maybe even pedantic—note/clarification: you can use this framework to "criticize a decision" in the sense that you can analyze/dispute the advantages claimed in favor of a decision, but you can also use the framework to make the assumptions of your disadvantages explicit (i.e., for purposes of reasoning transparency). So, the former is like using a blueprint to figure out how best to tear down a house, whereas the latter is like using a blueprint to build a house, where the houses are advantages or disadvantages (as opposed to the overall decision, which is more like the neighborhood of good and bad houses).

Agree (I think) with Khorton below, a worked example would be great.

I did include some example applications in the long introduction post (see my response to Khorton); I was worried that trying to include an example in this version might make it too long and thus lead to a high bounce rate… but perhaps I should have made it clear that I do have some applications in the old post.

Here is the first example (with updated terminology):

Consider lobbying for some policy change in a developing country—for example, on tobacco policy. Suppose that the proposal is to fund an advocacy campaign that would push for tighter controls on cigarettes, with the primary claimed advantage being “it will (increase the likelihood of passing legislation that will) reduce the mortality caused by smoking.” To evaluate this advantage, you would likely face questions such as:

  1. Counterfactuality: What would happen without this intervention? (Imagine for example that someone claims the campaign is likely to work because there is a “growing wave of support” for the reform: this might mean that the reform—or a slightly less strong version of the reform—already has a decent chance of passing. As part of this, it may be the case that the advocacy campaign will already receive sufficient funding.)
  2. Implementation: Do we actually have the necessary funding and can we actually meet the timeline outlined by the plan? (For example, are there any restrictions on foreign funding that have not been accounted for?)
  3. Linkage: Supposing that the plan is implemented (or, for a given implementation of the plan), what is the resulting likelihood that the desired reform will be signed into law—and subsequently, how effective will the desired reform be in reducing mortality caused by smoking (which introduces a recursion of this framework).
  4. Significance (assuming a utilitarian moral framework): How does “reducing mortality caused by smoking” translate to changes in wellbeing? If one considers the goal to simply be reducing mortality caused by smoking, that might be achieved, but it’s not guaranteed that achieving that goal will lead to an increase in wellbeing, such as is more-directly measured by a metric like QALYs. (For example, it’s possible that there are other widespread environmental problems that significantly reduce the effect of smoking mortality reduction on QALYs.)

Where would unintended consequences fit into this?

E.g. if someone says:

"This plan would cause X, which is good. (Co) X would not occur without this plan, (I) We will be able to carry out the plan by doing Y, (L) the plan will cause X to occur, and (S) X is morally good."

And I reply:

"This plan will also cause Z, which is morally bad, and outweights the benefit of X"

Which of the 4 categories of claim am I attacking? Is it 'implementation'?

"This plan will also cause Z, which is morally bad" is its own disadvantage/con.

"... and outweighs the benefit of X" relates to the caveat listed in footnote 3: you are no longer attacking/challenging the advantage itself ("this plan causes X"), but rather just redirecting towards a disadvantage. (Unless you are claiming something like "the benefits of X are not as strong as you suggested," in which case you're attacking it on significance.)

Curated and popular this week
Relevant opportunities