Shouldn’t we know better than to update in retrospect based on one highly uncertain datapoint?

We have a number of political data people in EA who thought donating to Flynn was a good investment early in the campaign cycle (later on I was hearing they thought it was no longer worth it). There was also good reason to believe Flynn could be high-impact if elected. Let’s not overthink this.

Thanks for writing this. While I don’t personally enjoy being featured, I appreciate the post as a Forum reader and former mod.

A few notes on my approach to donating, since I was quoted:

• Before choosing to donate, I spoke with two members of Carrick's campaign team about their plans and what the early donations would go toward, did some background reading on the district and Oregon politics, and looked over Carrick's campaign website and work history.
• My view wound up looking similar to Zach's (updated) view — Carrick's chances weren't great, but this still seemed like a strong opportunity. I didn't have an estimate for "marginal % increase in his chances per dollar"; had I been asked to make one, I think I'd have come out somewhat lower than ASB. But I still thought the value proposition was strong.
• I donated before ASB's post went up, and well before I knew that Protect Our Future would join in. Had I known that a PAC would spend millions, there's some chance I wouldn't have donated (my views would have changed, but it's hard to say exactly how).
  • It's arguable that I should have considered the chance that some wealthy donor would provide PAC support to Flynn. I'll take (dis)credit for that. But it honestly never crossed my mind; I'd just been on the phone with an obviously scrappy and overstretched campaign, and I saw a time-sensitive opportunity for marginal impact.
• Looking back, I wish I'd worded my comment more carefully. Rather than "I recommend this highly to people looking for impactful donations", I should have said "I highly recommend that people looking for impactful donations consider this as one option". Or just “I made a donation and I’m excited to see where this goes”, full stop.
  • I didn't expect or intend my comment to convince anyone to donate — I have no political experience or research background, and I didn't include any models or estimates! I mostly wrote the comment out of excitement.
  • But I did want to be open about my choice (I share all my donations online), and I did think it would be valuable for more people to consider donating (and of course, to do their own research and thinking beforehand).

Other thoughts:

• I was disappointed by some of the voting patterns I saw in the comments on ASB's post, including on Zach's comment. I was even more disappointed in the patterns on this post; I upvoted or strong-upvoted many of _pk's comments there, and reported one especially bad comment to the moderators.
• Looking back, I could have posted a bounty for the best argument against donating to the campaign. I did offer funding to the red-teaming contest, but they didn’t end up needing it, and it would have been too late for the election. Something to do next time I share a donation in public, maybe…
• I think that the “socially punished” sentence of Habryka's comment was wrong. I wouldn't expect any large-scale reputational hit* for anyone who argued against supporting Carrick, pointed out flaws in his work, etc., as long as the arguments were solid.
  • I do think low-quality support skates by in ways that low-quality criticism doesn’t, which is a problem, but a different problem.

*I won’t say “no reputational hit at all”, because thousands of people read the Forum and some of them would probably be annoyed. Public online discussion is rough.

I definitely agree that we need reflection on the questions, but I frankly think that this post needed a bit more basic fact checking.

But when the community is donating on the scale of $100K or more, particularly in novel ways, and particularly when many donors are driven by prominent EAs' enthusiasm, we should really have reason to believe that the intervention is effective. Before donating and encourage others to donate, we should share reason to believe that the intervention has a reasonable probability of success, rather than just the promise of great outcomes if it does succeed. 

Sure, but I think the first bar - expectation of impact conditional on success - was correct, and don't think you are disagreeing. And the second claim, that we need "reasonable probability of success," is a different and contentious claim, given all the discussion of hits-based giving, and the idea of expected value.

I almost think it's irresponsible for community leaders to say things like "I donated $5800 and recommend this highly to people looking for impactful individual donations" without (1) pointing to at least a back-of-the-envelope reason to believe that marginal donations have a reasonable probability of improving the outcome and (2) checking to make sure at least a couple people with relevant subject knowledge agree.

But 1) they did the BOTEC, if you read the post you linked to, and 2) yes, GAP and the various other people who focus on public policy and supporting the campaign personally were both working with plenty of people with subject matter expertise.

All of that said, we should certainly update based on the information gained from this loss, and I definitely think that more reflection is critical - but the reflexive take that "Carrick lost, so this was a bad decision" is unhelpful, and I'd welcome more specific thoughts on how and why "the community's decision-making process wasn't very grounded in donations' probability of tipping the election" - which seems plausible, but unclear, and far from the only thing that matters.

Epistemic status: very tired.

As others mentioned, this feels like too much of an update based on one data point. 

One of the largest advantages EAs running for office will have is their ability to fundraise from other EAs. I worry that skepticism of EAs in politics and/or slowness to act on time sensitive donation oppos will kneecap the success of future candidates. 

Big picture, I think the impact case was pretty solid. The US govt is enormously influential. It moves a lot of money, regulates important industries, has the largest military, and can uniquely affect x risk. Members of congress exert significant control over the govt. Senators more, president most. 

Having an extremely committed EA in govt seems worth A LOT to me. 

Raising some amount of money is essential to winning, no matter how much outside money is committed to a race. Campaigns need to hire staff, get on the ballot, and do other things that super PACs can't do. They also get much more favorable rates on TV ad buys, can make better ads, etc. "Hard money", i.e. that raised by campaigns by retail donors and governed by donor caps, is way more valuable than "soft money", i.e. independent expenditure made by super PACs. 

It seems clear to me that marginal hard dollars increase the odds of success, and it doesn't have to be that big of an increase for it to be a good bet in expected value terms. 

I would guess that almost no EAs donating to GiveWell charities really understand the evidence base and models going into the recommendation, but we outsource our thinking to people/orgs we trust. Obviously, there's way less of a track record with running EAs for office and a lot of uncertainty baked into politics. But the most experienced, aligned people in the political data science world were supportive of this particular race happening, and A LOT of thinking went into this decision. 

If you had to do it yourself, how would you go about a back-of-the-envelope calculation for estimating the impact of a Flynn donation?

Asking this question because I suspect that other people in the community won't actually do this, and since you are maybe one of the best-positioned people to do this since you seem interested in it.

Hey folks, good back and forth here. Just thought this story we just published on Protect Our Future and SBF might be interesting to the question about the value of a donation for Carrick and the cost-per-vote in this race. 

My back of the envelope. Back back back, like maybe even outhouse of the envelope. It's very hard to calculate marginal cost per vote, in part bc there's sort of an efficient markets thing going on w donations in some cases and not others. A senate race in Wyoming costs roughly the same as one in California, because the seat is just as valuable, despite vastly different numbers of votes. But activists getting worked up about a race can change the numbers, and in solid blue states it seems you can win a House seat more cheaply than in swing states, where you have to win a primary and a general and the national partisan orgs dgaf about the primary. 

: 

Approx 10m voters who matter in a Presidential campaign (i.e. the relevant pop of swing states). Takes about a billion to win. Naively this would suggest $100/vote, but multiply by 10x to $1000 for the marginal vote (iirc there IS some fancy econometrics way to estimate marginal dollar's impact better than this and I have seen it before; this is just dumdum math and I am dumdum). 

Do these numbers hold up for campaigns at other levels? Gonna set aside the Flynn campaign bc freakishly large amounts were spent. Normal House primary, maybe  1m spent. Maybe 100k votes cast in a blue district? Idk; it varies widely. I got this estimate from Ayanna Pressley's race, but it looks like in AOC's race only 30k were case, and 63k in Chris Pappas's primary race in a swing district. So, taking AOC as a sort of worst case,  30k votes cost 1m, ($33k each) assume the marginal vote is 10x that, it's $330 each. 

Or, thinking in terms of $ per election and ignoring numbers of votes, assuming 1m gets you a 50% chance of winning, $1=1/2m probability of winning election in a normal election.

But Flynn campaign cost more like $2000/vote (not marginal, avg) if reports of 8-figure SBF spend are to be believed. So under the same methodology this would imply ~20k per marginal vote.