All of Thomas Kwa's Comments + Replies

This seems right, thanks.

[Warning: long comment] Thanks for the pushback. I think converting to lives is good in other cases, especially if it's (a) useful for judging effectiveness, and (b) not used as a misleading rhetorical device [1].

The basic point I want to make is that all interventions have to pencil out. When donating, we are trying to maximize the good we create, not decide which superficially sounds better between the different strategies "empower beneficiaries to invest in their communities' infrastructure" and "use RCTs to choose lifesaving interventions" [2]. Lives a... (read more)

Just to remind everyone, 339,000 GBP in malaria nets is estimated by GiveWell to save around 61 lives, mostly young children. Therefore a 25% difference in effectiveness either way is 15 lives. A cost-effectiveness analysis is definitely required given what is at stake, even if the complexities of this project mean it is not taken as final.

Thanks. In addition to lots of general information about FTX, this helps answer some of my questions about FTX: it seems likely that FTX/Alameda were never massively profitable except for large bets on unsellable assets (anyone have better information on this?); even though they had large revenues maybe much of it was spent dubiously by SBF. And the various actions needed to maintain a web of lies indicate that Caroline Ellison and Nishad Singh, and very likely Gary Wang and Sam Trabucco (who dropped off the face of the earth at the time of the bankruptcy ... (read more)

I think someone should do an investigation much wider in scope than what happened at FTX, covering the entire causal chain from SBF first talking to EAs at MIT to the damage done to EA. Here are some questions I'm particularly curious about:

• Did SBF show signs of dishonesty early on at MIT? If so, why did he not have a negative reputation among the EAs there?
• To what extent did EA "create SBF"-- influence the values of SBF and others at FTX? Could a version of EA that placed more emphasis on integrity, diminishing returns to altruistic donations, or somethin

2-year update on infant outreach

To our knowledge, there have been no significant infant outreach efforts in the past two years. We are deeply saddened by this development, because by now there could have been two full generations of babies, including community builders who would go on to attract even more talent. However, one silver lining is that no large-scale financial fraud has been committed by EA infants.

We think the importance of infant outreach is higher than ever, and still largely endorse this post. However, given FTX events, there are a few chan... (read more)

This post is important and I agree with almost everything it says, but I do want to nitpick one crucial sentence:

There may well come a day when humanity would tear apart a thousand suns in order to prevent a single untimely death.

I think it is unlikely that we should ever pay the price of a thousand suns to prevent one death, because tradeoffs will always exist. The same resources used to prevent that death could support trillions upon trillions of sentient beings at utopic living standards for billions of years, either biologically or in simulation. The o... (read more)

My study of the monkeys and infants, i.e. my analysis of past wars, suggested an annual extinction risk from wars of 6.36*10^-14, which is still 1.07 % (= 5.93*10^-12/(5.53*10^-10)) of my best guess.

The fact that one model of one process gives a low number doesn't mean the true number is within a couple orders of magnitude of that. Modeling mortgage-backed security risk in 2007 using a Gaussian copula gives an astronomically low estimate of something like 10^-200, even though they did in fact default and cause the financial crisis. If the bankers adjusted ... (read more)

Don't have time to reply in depth, but here are some thoughts:

• If a risk estimate is used for EA cause prio, it should be our betting odds / subjectie probabilities, that is, average over our epistemic uncertainty. If from our point of view a risk is 10% likely to be >0.001%, and 90% likely to be ~0%, this lower bounds our betting odds at 0.0001%. It doesn't matter that it's more likely to be 0%.
• Statistics of human height are much better understood than nuclear war because we have billions of humans but no nuclear wars. The situation is more analogous to

Any probability as low as 5.93*10^-12 about something as difficult to model as the effects of nuclear war on human society seems extremely overconfident to me. Can you really make 1/5.93*10^-12 (170 billion) predictions about independent topics and expect to be wrong only once? Are you 99.99% [edit: fixed this number] sure that there is no unmodeled set of conditions under which civilizational collapse occurs quickly, which a nuclear war is at least 0.001% likely to cause? I think the minimum probabilities that one should have given these considerations is not much lower than the superforecasters' numbers.

There was likely no FTX polycule (a Manifold question resolved 15%) but I was aware that the FTX and Alameda CEOs were dating. I had gone to a couple of FTX events but try to avoid gossip, so my guess is that half of the well-connected EAs had heard gossip about this.

Being attention-getting and obnoxious probably paid off with slavery because abolition was tractable. But animal advocacy is different. I think a big question is whether he was being strategic, or just obnoxious by nature? If we put Benjamin Lay in 2000, would he start cage-free campaigns or become PETA? Or perhaps find some angle we're overlooking?

My comment is not an ad hominem. An ad hominem attack would be if someone is arguing point X and you distract from X by attacking their character. I was questioning only Remmelt's ability to distinguish good research from crankery, which is directly relevant to the job of an AISC organizer, especially because some AISC streams are about the work in question by Forrest Landry. I apologize if I was unintentionally making some broader character attack. Whether it's obnoxious is up to you to judge.

Crossposted from LessWrong.

Maybe I'm being cynical, but I'd give >30% that funders have declined to fund AI Safety Camp in its current form for some good reason. Has anyone written the case against? I know that AISC used to be good by talking to various colleagues, but I have no particular reason to believe in its current quality.

• MATS has steadily increased in quality over the past two years, and is now more prestigious than AISC. We also have Astra, and people who go directly to residencies at OpenAI, Anthropic, etc. One should expect that AISC doesn't

Surely as their gold standard “career change” pin-up story, they could find a higher EV career change.

You're assuming that the EV of switches from global health to biosecurity is lower than the EV of switching from something else to biosecurity. Even though global health is better than most cause areas, this could be false in practice for at least two reasons

• If the impact of biosecurity careers is many times higher than the impact of global health, and people currently in global health are slightly more talented, altruistic, or hardworking.
• If people currently in global health are not doing the most effective global health interventions.

This article just made HN. It's a report saying that 39 of 50 top offsetting programs are likely junk, 8 "look problematic", and 3 lack sufficient information, with none being found good.

I think most climate people are very suspicious of charities like this, rather than or in addition to not believing in ethical offsetting. See this Wendover Productions video on problematic, non-counterfactual, and outright fraudulent climate offsets. I myself am not confident that CATF offsets are good and would need to do a bunch of investigation, and most people are not willing to do this starting from, say, an 80% prior that CATF offsets are bad.

Upvoted. I don't agree with all of these takes but they seem valuable and underappreciated.

But with no evidence, just your guesses. IMO we should wait until things shake out and even then the evidence will require lots of careful interpretation. Also EA is 2/3 male, which means that even minor contributions of women to scandals could mean they cause proportionate harms.

I'm looking for AI safety projects with people with some amount of experience. I have 3/4 of a CS degree from Caltech, one year at MIRI, and have finished the WMLB and ARENA bootcamps. I'm most excited about activation engineering, but willing to do anything that builds research and engineering skill.

If you've published 2 papers in top ML conferences or have a PhD in something CS related, and are interested in working with me, send me a DM.

Upvoted for making an actual calculation with reasonable numbers.

Is there any evidence for this claim? One can speculate about how average personality gender differences would affect p(scandal), but you've just cited two cases where women caused huge harms, which seems to argue neutrally or against you.

Who tends to be clean?

With all the scandals in the last year or two, has anyone looked at which recruitment sources are least likely to produce someone extremely net negative in direct impact or to the community (i.e. a justified scandal)? Maybe this should inform outreach efforts.

In addition to everything mentioned so far, there's the information and retributive justice effect of the public exposé, which can be positive. As long as it doesn't devolve into a witch hunt, we want to discourage people from using EA resources and trust in the ways Nonlinear did, and this only works if it's public. If this isn't big enough, think about the possibility of preventing FTX. (I don't know if the actual fraud was preventable, but negative aspects of SBF's character and the lack of separation between FTX and Alameda could have been well substantiated and made public. Just the reputation of EAs doing due diligence here could have prevented a lot of harm.)

When I wrote the comment, it wasn't clear to me what the aim of the post was, and I thought Rockwell's reply clarified this. I just misinterpreted "accomplishments" at the top as being about impact rather than community. So I'm now glad this post exists, though citing metrics still bothers me a bit.

Crop yields are extremely low in much of Africa so my guess is there's potential for farmed animals to be fed while keeping constant or even decreasing land use.

Some questions I would be interested in:

• Where will Sub-Saharan Africa stand in terms of meat consumption and especially chicken consumption as the standard of living increases? When/if Nigeria hits China's current GDP per capita of $12.5k do we expect more or less meat consumption than China has?
• Then there's the welfare side. As countries get richer we have seen animal welfare get worse due to factory farming, then perhaps slightly better due to ability to afford animal welfare measures. Will we see the same trajectory in Africa, or should we expect something different like the ability to "leapfrog" to some of the worst animal agriculture practices, or even past them?

What object-level achievements has EA NYC had, in terms of money moved, talent found and placed in organizations, policy and technical achievements by those organizations, etc.? To be a bit blunt, the EA community and community members are not the point of EA, impact is the point.

I think funding is a bottleneck. Everything I've heard suggests the funding environment is really tight: CAIS is not hiring due to lack of funding. FAR is only hiring one RE in the next few months due to lack of funding. Less than half of this round of MATS scholars were funded for independent research. I think this is because there are not really 5-10 EA funders able to fund at large scale, just OP and SFF; OP is spending less than they were pre-FTX. At LTFF the bar is high, LTFF's future is uncertain, and they tend not to make huge grants anyway. So secu... (read more)

This currently has +154 karma on EA Forum and only +24 on LW, with similar exposure on each site, so I think it's fair to say that the reception is positive here and negative on LW. Maybe it's worth thinking about why.

Manifold markets related to this:
 

Fixed

What do you mean by "compassionate"?

Should the EA Forum team stop optimizing for engagement?
I heard that the EA forum team tries to optimize the forum for engagement (tests features to see if they improve engagement). There are positives to this, but on net it worries me. Taken to the extreme, this is a destructive practice, as it would

• normalize and encourage clickbait;
• cause thoughtful comments to be replaced by louder and more abundant voices (for a constant time spent thinking, you can post either 1 thoughtful comment or several hasty comments. Measuring session length fixes this but adds

I was thinking of reasons why I feel like I get less value from EA Forum. But this is not the same as reasons EAF might be declining in quality. So the original list would miss more insidious (to me) mechanisms by which EAF could actually be getting worse. For example I often read something like "EA Forum keeps accumulating more culture/jargon; this is questionably useful, but posts not using the EA dialect are received increasingly poorly." There are probably more that I can't think of, and it's harder for me to judge these...

Yeah, I don't think it's possible for controlled substances due to the tighter regulation.

The epistemic spot checker could also notice flaws in reasoning; I think Rohin Shah has done this well.

Reasonably often (maybe once or twice a month?) I see fairly highly upvoted posts that I think are basically wrong in something like "how they are reasoning", which I'll call epistemics. In particular, I think these are cases where it is pretty clear that the argument is wrong, and that this determination can be made using only knowledge that the author probably had (so it is more about reasoning correctly given a base of knowledge).

This is also on LessWrong.

Note that people in US/UK and presumably other places can buy drugs on the grey market (e.g. here) for less than standard prices. Although I wouldn't trust these 100%, they should be fairly safe because they're certified in other countries like India; gwern wrote about this here for modafinil and the basic analysis seems to hold for many antidepressants. The shipping times advertised are fairly long but potentially still less hassle than waiting for a doctor's appointment for each one.

Thanks. It looks reassuring that the correlations aren't as large as I thought. (How much variance is in the first principal component in log odds space though?) And yes, I now think the arguments I had weren't so much for arithmetic mean as against total independence / geometric mean, so I'll edit my comment to reflect that.

The main assumption of this post seems to be that, not only are the true values of the parameters independent, but a given person's estimates of stages are independent. This is a judgment call I'm weakly against.

Suppose you put equal weight on the opinions of Aida and Bjorn. Aida gives 10% for each of the 6 stages, and Bjorn gives 99%, so that Aida has an overall x-risk probability of 10^-6 and Bjorn has around 94%.

• If you just take the arithmetic mean between their overall estimates, it's like saying "we might be in worlds where Aida is correct, or worlds

Probabilities of probabilities can make sense if you specify what they're over. Say the first level is the difficulty of the alignment problem, and the second one is our actions. The betting odds on doom collapse, but you can still say meaningful things, e.g. if we think there's a 50% chance alignment is 1% x-risk and a 50% chance it's 99% x-risk, then the tractability is probably low either way (e.g. if you think the success curve is logistic in effort).

Surely reducing the number of players, making it more likely that US entities develop AGI (who might be more or less careful, more or less competent, etc. than Chinese entities), and (perhaps) increasing conflict all matter for alignment? There are several factors here that push in opposite directions, and this comment is not an argument for why the sum is zero to negative.

(I'm helping Vivek and Nate run the consequentialist cognition MATS stream)

Yes, both of those are correct. The formatting got screwed up in a conversion, and should be fixed soon.

In the future, you could send Vivek or me a DM to contact our project specifically. I don't know what the official channel for general questions about MATS is.