When modeling with uncertainty we often care about the expected value of our result. In CEAs, in particular, we often try to estimate . This is different from both  and  (which are also different from each other). [1] The goal of this post is to make this clear.

One way to simplify this is to assume that the cost is constant. So we only have uncertainty about the effect. We will also assume at first that the effect can only be one of two values, say either 1 QALY or 10 QALYs with equal probability.

Expected Value is defined as the weighted average of all possible values, where the weights are the probabilities associated with these values. In math notation, for a random variable ,

where  are all of the possible values of .[2] For non-discrete distributions, like a normal distribution, we'll change the sum with an integral.

Coming back to the example above, we seek the expected value of effect over cost. As the cost is constant, say  dollars, we only have two possible values:

In this case we do have , but as we'll soon see that's only because the cost is constant. What about ?

which is not , a smaller amount.

The point is that generally  In fact, we always have  with equality if and only if  is constant.[3]

Another common and useful example is when  is lognormally distributed with parameters . That means, by definition, that   is normally distributed with expected value and variance  respectively. The expected value of  itself is a slightly more complicated expression:

Now the fun part:  is also lognormally distributed! That's because . Its parameters are  (why?) and so we get 

In fact, we see that the ratio between these values is 

  1. ^

    See Probability distributions of Cost-Effectiveness can be misleading for relevant discussion. There are arguably reasons to care about the two alternatives  or  rather than  , which are left for a future post.

  2. ^

    One way to imagine this is that if we sample  many times we will observe each possible value  roughly  of the times. So the expected value would indeed generally be approximately the average value of many independent samples.

  3. ^

55

3
0

Reactions

3
0
Comments14


Sorted by Click to highlight new comments since:
Bella
16
0
0
1

Commenting to help out any other people confused by the mathematical notation, because I couldn't find this out with Google (but ChatGPT got it for me):

Expectation: In probability and statistics, 'E' is often used to denote the expected value of a random variable. For example, E(X) would represent the expected value of the random variable X.

(Of course, given that I didn't know that, I have no hope of following the entire post, but at least I now understand roughly what the claim is)

Great post! A particular issue is that E(cost/effect) is infinite or undefined if you have a non-zero probability that the effect is 0. This is very commonly the case.

Another interesting point, highlighted by your log normal example, is that higher variance will tend to increase the difference between E(1/X) and 1/E(X).

E[effect/cost] will also inflate the cost-effectiveness, giving too much weight to cases where you spend little and too little weight to your costs (and opportunity costs) when you spend a lot. If there’s a 90% chance the costs are astronomical and there's no impact or it's net negative, the other 10% could still make it "look" cost-effective. The whole project could have negative expected effects (negative E[effect]), but positive E[effect/cost]. That would not be a project worth supporting.

You should usually be estimating E[costs]/E[effects] or E[effects]/E[costs], not the expected values of ratios.

Have you figured out exactly when "E[costs]/E[effects] or E[effects]/E[costs]" is called for? I have historically agreed with the point you are making, but my beliefs have been shaken recently. Here's an example that has made me think twice:

You are donating $100 to a malaria charity and can choose between charity A and B. Charity A gets bednets for $1 each. Charity B does not yet know the cost of its bednets, but they will cost either $0.50 or $1.50 with equal probability.

Donating to charity A has a value of 100 bednets. Donating to charity B has expected value 133 bednets (equal chance of buying 200 or 66). But "E[costs]/E[effects] or E[effects]/E[costs]" is the same for each charity. In this case, E[effect/cost] seems like the right metric.

So is, the difference the fact that total costs are fixed? Someone deciding whether to start an organization or to commit to fully-funding a new intervention would have to contend with variable, unknown total costs.

Is it because funding charity A involves buying 100 "shares" in an intervention, and funding charity B involves buying either 200 or 66  "shares", which "E[costs]/E[effects] or E[effects]/E[costs]" fails to capture?

If you set costs=$100 as constant in this case, then

E[effects/costs] = E[effects]/costs = E[effects]/E[costs],

and both are right.

The cases where it matters are the ones where you don't know how much you'll spend, including if you're starting or running a charity, how much your charity will spend. For example, depending on your outputs, impact and updated expectations for cost-effectiveness, you'd stop taking donations and shut down.

If you wanted specifically to buy exactly 100 bednets, then committing to B would be worse, because you'll spend more than $100 in expectation, and each extra expected dollar could have bought another bednet from A. This would be more relevant from the perspective of a charity that doesn't know how much it'll need to spend to reach a specific fixed binary goal, like a government policy change. But, the ratios of expected values still seem right here.

I'm not sure there are any cases where E[costs]/E[effects] or E[effects]/E[costs] gives the wrong answer in practical applications for resource allocation, if costs is what you'll spend on the intervention. E[effects] is what you're trying to maximize, and you can get that by multiplying E[effects]/E[costs] and E[costs]. E[effects/costs] won't in general give you E[effects] when you multiply by E[costs].

If you set costs=$100 as constant in this case, then

E[effects/costs] = E[effects]/costs = E[effects]/E[costs],

and both are right.

Ah yes, I see that.

Having now read the post that Lorenzo recommended below, I'm coming round to the majority view that the key question is "how much good could we expect from a fixed unit of cost?".

I think in this thread there are two ways of defining costs:

  1. Michael considers the cost as the total amount spent
  2. Stan suggests a case where the cost is the amount needed to be spent per unit of intervention

I think this is the major source of disagreement here, right?

This discussion resembles the observation that the cost-effectiveness ratio should mostly be used in the margin. That is, in the end we care about something like  and when we decide where to spend the next dollar we should compute the derivatives with respect to that extra resource and choose the intervention which maximizes that increased value. 

I thought about this a bit and have edited this post from last year. I'm curious if you find it useful!

There's also lots of discussion in the comments on that post about why E[effect/cost] is better than E[effects]/E[costs] (which I originally argued for) according to most commenters (which I now agree with).

Thank you. I'd been drafting a very similar post of my own!!

It's probably still worth posting! E.g. it seems that @MichaelStJules and commenters on my post would disagree on defaulting to E[effect/cost] vs E[effects]/E[costs]

Things to add:

  1. Graphical example
  2. Explicit discussion of GiveWell's use of cost/effect 
  3. Real-world examples where this causes a significant error 
  4. Expand on the lognormal example, discuss cases where the variance is in the same order of magnitude as the expected value
  5. What this says about how we should model uncertainty

From my experience last year writing the post you linked to (which I now just edited), I would really dumb things down and try to make the graphical example as simple as possible for people with ~high-school math knowledge (like me)

Several smart people I asked for advice about that post seemed to find reasoning about this not trivial, gave somewhat contradicting answers, or thought that the solution I had thought of was too obvious to post, despite it being very wrong (as you explained in the comments last year, thank you!) Even after a year, tbh I'm not sure I "grok" it completely (see the appendix I added to the post. But I reassure myself with the fact that costs very rarely span across orders of magnitude, and in those cases, I could try to model E(value) directly rather than E(value/cost) as a proxy).

Thanks! Yeah, I totally agree - the topic is surprisingly delicate and nonintuitive, and my examples above are too technical. 

By the way, I'd love it if other people would write posts that make the exact same point but better! (or for different audiences). 

I think most people who'd benefit from reading this will bounce off the equations above. Here is a concrete example that I think is easy to understand.

Consider a variable X that can be 1 or 2 with equal probability.

E(X) = (1+2)/2 = 1.5

So 1/E(X) = 1/1.5 = 0.667

Whereas, E(1/X) = (1/1 + 1/2)/2 = 0.75

So sometimes 1/E(X) does not equal E(1/X).

Curated and popular this week
 ·  · 12m read
 · 
Economic growth is a unique field, because it is relevant to both the global development side of EA and the AI side of EA. Global development policy can be informed by models that offer helpful diagnostics into the drivers of growth, while growth models can also inform us about how AI progress will affect society. My friend asked me to create a growth theory reading list for an average EA who is interested in applying growth theory to EA concerns. This is my list. (It's shorter and more balanced between AI/GHD than this list) I hope it helps anyone who wants to dig into growth questions themselves. These papers require a fair amount of mathematical maturity. If you don't feel confident about your math, I encourage you to start with Jones 2016 to get a really strong grounding in the facts of growth, with some explanations in words for how growth economists think about fitting them into theories. Basics of growth These two papers cover the foundations of growth theory. They aren't strictly essential for understanding the other papers, but they're helpful and likely where you should start if you have no background in growth. Jones 2016 Sociologically, growth theory is all about finding facts that beg to be explained. For half a century, growth theory was almost singularly oriented around explaining the "Kaldor facts" of growth. These facts organize what theories are entertained, even though they cannot actually validate a theory – after all, a totally incorrect theory could arrive at the right answer by chance. In this way, growth theorists are engaged in detective work; they try to piece together the stories that make sense given the facts, making leaps when they have to. This places the facts of growth squarely in the center of theorizing, and Jones 2016 is the most comprehensive treatment of those facts, with accessible descriptions of how growth models try to represent those facts. You will notice that I recommend more than a few papers by Chad Jones in this
LintzA
 ·  · 15m read
 · 
Introduction Several developments over the past few months should cause you to re-evaluate what you are doing. These include: 1. Updates toward short timelines 2. The Trump presidency 3. The o1 (inference-time compute scaling) paradigm 4. Deepseek 5. Stargate/AI datacenter spending 6. Increased internal deployment 7. Absence of AI x-risk/safety considerations in mainstream AI discourse Taken together, these are enough to render many existing AI governance strategies obsolete (and probably some technical safety strategies too). There's a good chance we're entering crunch time and that should absolutely affect your theory of change and what you plan to work on. In this piece I try to give a quick summary of these developments and think through the broader implications these have for AI safety. At the end of the piece I give some quick initial thoughts on how these developments affect what safety-concerned folks should be prioritizing. These are early days and I expect many of my takes will shift, look forward to discussing in the comments!  Implications of recent developments Updates toward short timelines There’s general agreement that timelines are likely to be far shorter than most expected. Both Sam Altman and Dario Amodei have recently said they expect AGI within the next 3 years. Anecdotally, nearly everyone I know or have heard of who was expecting longer timelines has updated significantly toward short timelines (<5 years). E.g. Ajeya’s median estimate is that 99% of fully-remote jobs will be automatable in roughly 6-8 years, 5+ years earlier than her 2023 estimate. On a quick look, prediction markets seem to have shifted to short timelines (e.g. Metaculus[1] & Manifold appear to have roughly 2030 median timelines to AGI, though haven’t moved dramatically in recent months). We’ve consistently seen performance on benchmarks far exceed what most predicted. Most recently, Epoch was surprised to see OpenAI’s o3 model achieve 25% on its Frontier Math
Omnizoid
 ·  · 5m read
 · 
Edit 1/29: Funding is back, baby!  Crossposted from my blog.   (This could end up being the most important thing I’ve ever written. Please like and restack it—if you have a big blog, please write about it). A mother holds her sick baby to her chest. She knows he doesn’t have long to live. She hears him coughing—those body-wracking coughs—that expel mucus and phlegm, leaving him desperately gasping for air. He is just a few months old. And yet that’s how old he will be when he dies. The aforementioned scene is likely to become increasingly common in the coming years. Fortunately, there is still hope. Trump recently signed an executive order shutting off almost all foreign aid. Most terrifyingly, this included shutting off the PEPFAR program—the single most successful foreign aid program in my lifetime. PEPFAR provides treatment and prevention of HIV and AIDS—it has saved about 25 million people since its implementation in 2001, despite only taking less than 0.1% of the federal budget. Every single day that it is operative, PEPFAR supports: > * More than 222,000 people on treatment in the program collecting ARVs to stay healthy; > * More than 224,000 HIV tests, newly diagnosing 4,374 people with HIV – 10% of whom are pregnant women attending antenatal clinic visits; > * Services for 17,695 orphans and vulnerable children impacted by HIV; > * 7,163 cervical cancer screenings, newly diagnosing 363 women with cervical cancer or pre-cancerous lesions, and treating 324 women with positive cervical cancer results; > * Care and support for 3,618 women experiencing gender-based violence, including 779 women who experienced sexual violence. The most important thing PEPFAR does is provide life-saving anti-retroviral treatments to millions of victims of HIV. More than 20 million people living with HIV globally depend on daily anti-retrovirals, including over half a million children. These children, facing a deadly illness in desperately poor countries, are now going
Relevant opportunities