All of Habryka's Comments + Replies

This is an extremely rich guy who isn't donating any of his money.

FWIW, I totally don't consider "donating" a necessary component of taking effective altruistic action. Most charities seem much less effective than the most effective for-profit organizations, and most of the good in the world seems achieved by for-profit companies. 

I don't have a particularly strong take on Bryan Johnson, but using "donations" as a proxy seems pretty bad to me.

Less than a year ago Deepmind and Google Brain were two separate companies (both making cutting-edge contributions to AI development). My guess is if you broke off Deepmind from Google you would now just pretty quickly get competition between Deepmind and Google Brain (and more broadly just make the situation around slowing things down a more multilateral situation).

But more concretely, anti-trust action makes all kinds of coordination harder. After an anti-trust action that destroyed billions of dollars in economic value, the ability to get people in the same room and even consider coordinating goes down a lot, since that action itself might invite further anti-trust action.

Huh, fwiw I thought this proposal would increase AI risk, since it would increase competitive dynamics (and generally make coordinating on slowing down harder). I at least didn't read this post as x-risk motivated (though I admit I was confused what it's primary motivation was).

Yeah, that's a decent link. I do think this comment is more about whether anti-recommendations for organizations should be held to a similar standard. My comment also included some criticisms of Sean personally, which I think do also make sense to treat separately, though at least I definitely intend to also try to debias my statements about individuals after my experiences with SBF in-particular on this dimension.

Hmm, I agree that there was some aggression here, but I felt like Sean was the person who first brought up direct criticism of a specific person, and very harsh one at that (harsher than mine I think). 

Like, Sean's comment basically said "I think it was directly Bostrom's fault that FHI died a slow painful death, and this could have been avoided with the injection of just a bit of competence in the relevant domain". My comment is more specific, but I don't really see it as harsher. I also have a prior to not go into critiques of individual people, but that's what Sean did in this context (of course Bostrom's judgement is relevant, but I think in that case so is Sean's).

Pushback (in the form of arguments) is totally reasonable! It seems very normal that if someone is arguing for some collective path of action, using non-shared assumptions, that there is pushback. 

The thing that feels weirder is to invoke social censure, or to insist on pushback when someone is talking about their own beliefs and not clearly advocating for some collective path of action. I really don't think it's common for people to push back when someone is expressing some personal belief of theirs that is only affecting their own actions. 

In t... (read more)

This also roughly matches my impression. I do think I would prefer the EA community to either go towards more centralized governance or less centralized governance in the relevant way, but I agree that given how things are, the EA Forum team has less leeway with moderation than the LW team. 

 I think this might be one of the LTFF writeups Oli mentions (apologies if wrong), and seems like a good place to start

Yep, that's the one I was thinking about. I've changed my mind on some of the things in that section in the (many) years since I wrote it, but it still seems like a decent starting point.

This thread doesn't feel great for this, though CSER is an organization for which I do really wish more people shared their assessments. Also happy to have a call if your curiosity extends that far, and you would be welcome to write up the things that I say in that call publicly (though of course that's a lot of work and I don't think you have any obligation to do so). 

Thanks Sean. I think this is a good comment and I think makes me understand your perspective better. 

I do think we obviously have large worldview differences here, that seem maybe worth exploring at some point, but this comment (as well as some private conversations sparked by these comments with others at FHI) made me feel more sympathetic to the perspective of "there is some history-rewriting happening that seems scary, where the university gets portrayed as this kind of boogeyman, and while it does seem the university did some unreasonable-seeming ... (read more)

You made hostile claims that weren't following on from prior discussion,^[1] and in my view nasty and personal insinuations as well, and didn't have anything to back it up. 

This seems relatively straightforwardly false. In as much as Sean is making claims about the right strategy to follow for FHI, and claiming that the errors at FHI were straightforwardly Bostrom's fault and attributable to 'garden variety incompetence', the degree of historical success of the strategies that Sean seems to be advocating for is of course relevant in assessing whet... (read more)

Yeah, I agree this is a real dynamic. It doesn't sound unreasonable for me to have a standard link that l link to if I criticize people on here that makes it salient that I am aspiring to be less asymmetric in the information I share (I do think the norms are already pretty different over on LW, where if anything I think criticism is a bit less scrutinized than praise, so its not like this is a totally alien set of norms).

I don't understand. I do not consider myself to be under the obligation that all negative takes I share about an organization must be accompanied by a full case for why I think those are justified.

Similar to how it would IMO be crazy to request people to justify that all positive comments about an organization must be accompanied by full justifications for ones judgement.

I have written about my feelings about CSER and Leverhulme some in the past (one of my old LTFF writeups for example includes a bunch of more detailed models I have of CSER). I have defini... (read more)

i am very glad that Bostrom chose the path he chose with FHI in contrast to the path you seem to have chosen with CSER and Leverhulme, so I am inclined to not trust your take here too much (though your closeness and direct experience of course is important and relevant here and makes up for some of my a-priori skepticism). Where FHI successfully continued to produce great intellectual work, I have seen other organizations in similar positions easily fall prey to the demand and pressures and become a hollow shell of political correctness and vapid ideas.

It ... (read more)

Thank you Will! This is very much the kind of reflection and updates that I was hoping to see from you and other leaders in EA for a while.

I do hope that the momentum for translating these reflections into changes within the EA community is not completely gone given the ~1.5 years that have passed since the FTX collapse, but something like this feels like a solid component of a post-FTX response. 

I disagree with a bunch of object-level takes you express here, but your reflections seem genuine and productive and I feel like me and others can engage with them in good faith. I am grateful for that.

Yeah, I think just buying Twitter to steer the narrative seems quite bad. But like, I have spent a large fraction of my career trying to think of mechanism design for discussion and social media platforms and so my relation to Twitter is I think a pretty healthy "I think I see lots of ways in which you could make this platform much more sanity-promoting" in a way that isn't about just spreading my memes and ideologies. 

Will has somewhat less of that background, and I think would have less justified confidence in his ability to actually make the platform better from a general sanity perspective, though still seems pretty plausible to me he saw or sees genuine ways to make the platform better for humanity.

I am the author of the linked post that DPiepgrass was commenting on: https://www.lesswrong.com/posts/HCAyiuZe9wz8tG6EF/my-tentative-best-guess-on-how-eas-and-rationalists

This link's hypothesis is about people just trying to fit in―but SBF seemed not to try to fit in to his peer group! He engaged in a series of reckless and fraudulent behaviors that none of his peers seemed to want.

(Author of the post) My model is that Sam had some initial tendencies for reckless behavior and bullet-biting, and those were then greatly exacerbated via evaporative cooling dynamics at FTX. 

It sounds like SBF drove away everyone who couldn't stand his methods until only people who tolerated him were left. That's a pretty different way of m

Do you have links to people being very worried about gray goo stuff?

(Also, the post you link to makes this clear, but this was a prediction from when Eliezer was a teenager, or just turned 20, which does not make for a particularly good comparison, IMO)

Also, I don't think "completely uninvestigated" is a correct characterization -- they were investigated enough to be presented to a grand jury, which indicted SBF for campaign-finance violations. Federal prosecutors do not generally indict without a pretty good investigation first, especially in high-profile cases. I think we have a pretty decent idea of what he did (see pp. 18-22 of the prosecution's sentencing memo). Moreover, Salame and Singh -- who don't have extradition-related issues -- pled guilty to campaign-finance violations.

Oh, that is interesti... (read more)

I... think I will continue describing this as "weird" though it makes sense that as a lawyer it's not that weird to you.

It feels very off to have a bunch of crimes uninvestigated because someone fled the country and then was extradited, and I am pretty confused why the Bahamas cooperated with Sam here (my guess is that it's a case of political corruption, though I can also imagine other reasons). It's not like the Bahamas had anything obvious to gain from Sam not being convicted of the campaign finance violations.

This is great!

Note that this list is not comprehensive. In-particular it doesn't go into detail on any of the campaign finance violations and other policy-stuff that Sam was involved in, which I think never ended up investigated due to a kind of weird agreement with the Bahamas authorities. But during the trial we did hear some pretty clear evidence of at least campaign finance violations (and my guess is there would be a bunch more if one kept digging, as well as stuff that wouldn't necessarily be crimes but stuff I think most people here would still consider highly unethical).

It seems to me that a case study of how exactly FTX occurred, and where things failed, would be among one of the best things to use to figure out what thing to do instead. 

Currently the majority of people who have an interest in this are blocked by not really knowing what worked and didn't work in the FTX case, and so probably will have trouble arguing compellingly for any alternative, and also lack some of the most crucial data. My guess is you might have the relevant information from informal conversations, but most don't. 

I do think also just ... (read more)

My current sense is that there is no motivation to find an alternative because people mistakenly think it works fine enough and so there is no need to try to find something better (and also in the absence of an investigation and clear arguments about why the rumor thing doesn't work, people probably think they can't really be blamed if the strategy fails again)

I think investing in FTX was genuinely a good idea, if you were a profit maximizer, even if you strongly suspected the fraud. As Jason says, as an investor losing money due to fraud isn't any worse than losing money because a company fails to otherwise be profitable, so even assigning 20%-30% probability to fraud for a high-risk investment like FTX where you are expecting >2x returns in a short number of years will not make a huge difference to your bottomline.

In many ways you should expect being the kind of person who is willing to commit fraud to be p... (read more)

I'm not the person quoted, but I agree with this part, and some of the reasons for why I expect the results of an investigation like this to be boring aren’t based on any private or confidential information, so perhaps worth sharing.

One key reason: I think rumor mills are not very effective fraud detection mechanisms.

Huh, the same reason you cite for why you are not interested in doing an investigation is one of the key reasons why I want an investigation. 

It seems to me that current EA leadership is basically planning to continue a "ou... (read more)

Becca, Nicole and Max all stand out as people who I think burned out trying to make things go better around FTX stuff. 

Also Claire leaving her position worsened my expectations of how much Open Phil will do things that seem bad. Alexander also seems substantially worse than Holden on this dimension. I think Holden was on the way out anyways, but my sense was Claire found the FTX-adjacent work very stressful and that played a role in her leaving (I don't thinks she agrees with me on many of these issues, but I nevertheless trusted her decision-making more than others in the space).

Yeah, I think this would only make sense if you would somehow end up majorly shaping the algorithms and structure of Twitter. I don't think just being a shareholder really does much here.

Thanks, that's very helpful, though if you think it's mostly because the PR cost has already been paid, then that does provide little solace under my worldview. Let's assume the PR costs were still ongoing, do you think it would have then flipped the decision?

FWIW, buying Twitter still seems plausibly like a good idea to me. It sure seems to be the single place that is most shaping public opinion on a large number of topics I care a lot about (like AI x-risk attitudes), and making that go better seems worth a lot.

I don't particularly feel like my knowledge here is confidential, it would just take a bunch of inferential distance to cross. I do have some confidential information, but it doesn't feel that load-bearing to me. 

This dialogue has a bit of a flavor of the kind of thing I am worried about: https://www.lesswrong.com/posts/vFqa8DZCuhyrbSnyx/integrity-in-ai-governance-and-advocacy?revision=1.0.0 

I think this captures some of what I mean, though my model is also that the "Maximally naive" view is not very stable, in that if you are being "maximally naive" you do often end up just lying to people (because the predictable benefits from lying to people outweigh the predictable costs in that moment). 

I do think a combination of being "maximally naive" combined with strong norms against deception and in favor of honesty can work, though in-general people want good reasons for following norms, and arguing for honesty requires some non-naive reasoning.

My understanding is that UK law and state law whistleblower protections are extremely weak and only cover knowledge of literal and usually substantial crimes (including in California). I don't think any legally-mandated whistleblower protections make much of a difference for the kind of thing that EAs are likely to encounter. 

I checked the state of the law in the FTX case, and unless someone knew specifically of clear fraud going on, they would have not been protected, which seems like it makes them mostly useless for things we care about. They also w... (read more)

Publishing it probably wouldn't make sense at any other time, so Zach may have been stuck between being rushed into publishing it too early or not responding to the public-interest event at all.

Yeah, agree, that makes sense. I do think it was the wrong call, but I can understand the perceived urgency.

Also, it seems unlikely he booked six weeks off right after SBF's sentencing for that reason.

Totally, to be clear, I think it's totally fine for Zack to take time off, and wasn't intending to comment on that at all. I was just responding to the (what I perceived to be a separate thread) of wanting to hold off on engaging until he had formed considered opinions.

I wouldn't read too much into the exact ordering in the EA Forum digest. At least if I was making such a digest I would mostly be busy filling it up at all, and it would feel unnecessarily nitpicky and stressful to me to be judged on even the relative ordering of the articles I put in there.

but I want to wait to publicly discuss my thoughts on these issues until I have the capacity to do so thoroughly and thoughtfully, rather than attempt to respond on the fly

You did speak publicly about them, in a large newspaper nonetheless: https://www.washingtonpost.com/opinions/2024/03/28/sam-bankman-fried-effective-truism-fraud

To be clear, I think it's still fine to take some time, but it does seem like you made claims that the EA community has engaged in successful investigation and reflection here, and so saying that you want to hold off on engaging u... (read more)

So, I think it's clear that a lot of leadership turnover has happened. However, my sense is that the kind of leadership turnover that has occurred is anti-correlated with what I would consider good. Most importantly, it seems to me that the people in EA leadership that I felt were often the most thoughtful about these issues took a step back from EA, often because EA didn't live up to their ethical standards, or because they burned out trying to affect change and this recent period has been very stressful (or burned out for other reasons, unrelated to tryi... (read more)

My sense is people are realizing this, based on the disagree-votes, but just for the record: No, the most important part of any FTX investigation is of course not that we prevent causing further harm to EA, lol. It's to prevent causing further harm to the world. Indeed, this kind of naive-consequentialist reasoning seems like one of the key components of how FTX happened in the first place.

I think in any world, including ones where EA leadership is dropping the ball or is likely to cause more future harm like FTX, it would be very surprising if they individually had not updated substantially. 

As an extreme illustrative example, really just intended to get the intuition across, imagine that some substantial fraction of EA leaders are involved in large scale fraud and continue to plan to do so (which to be clear, I don't have any evidence of), then of course the individuals would update a lot on FTX, but probably on the dimensions of "her... (read more)

My sense is the EV UK board mattered a good amount as well during this period, and Claire Zabel was also on the board during the relevant period (I do not know which board members Becca was thinking about in the above post, if any).

Is your sense that if the cost-effectiveness estimate had come back positive, but not overwhelmingly positive (let's say like a 70th percentile OP grant-dollar in the last year), that this would have flipped the decision? 

Given that not vetoing this grant made Alexander's top list of decisions he regrets, mostly because of the negative optics-aspects, I would be surprised if the cost-effectiveness estimate was actually a crux here.

As a critic of many institutions and organizations in EA, I agree with the above dynamic and would like people to be less nitpicky about this kind of thing (and I tried to live up to that virtue by publishing my own quite rough grant evaluations in my old Long Term Future Fund writeups)

What I mean by collaboration is "is willing to share any information with them and allow staff to speak freely". The key obstacle I have faced in trying to do investigations is that nobody is willing to talk or say anything that goes on the record by organizational policy, which of course makes this kind of thing very hard to pull off. 

I also think it would help a lot if CEA were to lend some credibility to the investigation. People don't want to repeat the same thing hundreds of times, and it would IMO be good for CEA/EV to put some social capital on the line to encourage people to talk to the investigators.

Yeah, that's fair. I'll edit it in. 

One final issue is who should sponsor an investigation of "what role EA played in FTX coming into existence"; given the linkages between EVF, EVF insiders, and SBF, EVF would be somewhere in the vicinity of last place on my ideal preference list.

I think CEA or EV collaborating with an external but still within-EA trusted organization seems like the best choice to me here. Hiring someone who is broadly known to be independent (like, IDK, you could choose someone from Rethink, or Tyler Cowen, or someone else in that kind of reference class), seems like a good idea.

I didn't intend the above to be a straightforward accusation of lying. My sense is that different people at CEA disagree here about how useful this investigation was for reform purposes (which I think is evidence of the claim being false, but not strong evidence of deception).

If I had to make a bet, I would say majority opinion at CEA on the question of "did this investigation obstruct or assist internal reform efforts at CEA (as opposed to e.g. address legal liability and reputational risks independent from their best guess about what appropriate reform i... (read more)

Yeah, I think you are pointing towards something real here. 

Like, I do think a thing that drove my reaction to this was a perspective in which it was obvious that most people in EA didn't literally actively participate in the FTX fraud. I have encountered very extreme and obviously wrong opinions about this in the public (the comment section of the WaPo article provides many examples of this), and there is some value in engaging with that. 

But I do think that is engaging with a position that is extremely shallow, and the mechanism of it seems lik... (read more)