All of elifland's Comments + Replies

Thanks for organizing this! Tentatively excited about work in this domain.

I do think that generating models/rationales is part of forecasting as it is commonly understood (including in EA circles), and certainly don't agree that forecasting by definition means that little effort was put into it!
Maybe the right place to draw the line between forecasting rationales and “just general research” is asking “is the model/rationale for the most part tightly linked to the numerical forecast?" If yes, it's forecasting, if not, it's something else. 

Thanks for clarifying! Would you consider OpenPhil worldview investigations repor... (read more)

Thanks for writing this up, and I'm excited about FutureSearch! I agree with most of this, but I'm not sure framing it as more in-depth forecasting is the most natural given how people generally use the word forecasting in EA circles (i.e. associated with Tetlock-style superforecasting, often aggregation of very part-time forecasters' views, etc.). It might be imo more natural to think of it as being a need for in-depth research, perhaps with a forecasting flavor. Here's part of a comment I left on a draft.

However, I kind of think the framing of the essay

Thanks Ozzie for chatting! A few notes reflecting on places I think my arguments in the conversation were weak:

1. It's unclear what short timelines would mean for AI-specific forecasting. If AI timelines are short it means you shouldn’t forecast non-AI things much, but it’s unclear what it means about forecasting AI stuff. There’s less time for effects to compound but you have more info and proximity to the most important decisions. It does discount non-AI forecasting a lot though, and some flavors of AI forecasting.
2. I also feel weird about the comparison I ma

Just chatted with @Ozzie Gooen about this and will hopefully release audio soon. I probably overstated a few things / gave a false impression of confidence in the parent in a few places (e.g., my tone was probably a little too harsh on non-AI-specific projects); hopefully the audio convo will give a more nuanced sense of my views. I'm also very interested in criticisms of my views and others sharing competing viewpoints.

Also want to emphasize the clarifications from my reply to Ozzie:

1. While I think it's valuable to share thoughts about the value of differen

Thanks Ozzie for sharing your thoughts!

A few things I want to clarify up front:

1. While I think it's valuable to share thoughts about the value of different types of work candidly, I am very appreciative of both people working on forecasting projects and grantmakers in the space for their work trying to make the world a better place (and am friendly with many of them). As I maybe should have made more obvious, I am myself affiliated with Samotsvety Forecasting, and Sage which has done several forecasting projects. And I'm also doing AI forecasting research at

All views are my own rather than those of any organizations/groups that I’m affiliated with. Trying to share my current views relatively bluntly. Note that I am often cynical about things I’m involved in. Thanks to Adam Binks for feedback. 

Edit: See also child comment for clarifications/updates. 

Edit 2:  I think the grantmaking program has different scope than I was expecting; see this comment by Benjamin for more.

Following some of the skeptical comments here, I figured it might be useful to quickly write up some personal takes on forecastin... (read more)

1. So in the multi-agent slowly-replacing case, I'd argue that individual decisions don't necessarily represent a voluntary decision on behalf of society (I'm imagining something like this scenario). In the misaligned power-seeking case, it seems obvious to me that this is involuntary. I agree that it technically could be a collective voluntary decision to hand over power more quickly, though (and in that case I'd be somewhat less against it).
2. I think emre's comment lays out the intuitive case for being careful / taking your time, as does Ryan's. I think the e

(edit: my point is basically the same as emre's)

I think there is very likely at some point going to be some sort of transition to a world where AIs are effectively in control. It seems worth it to slow down on the margin to try to shape this transition as best we can, especially slowing it down as we get closer to AGI and ASI. It would be surprising to me if making the transfer of power more voluntary/careful led to worse outcomes (or only led to slightly better outcomes such that the downsides of slowing down a bit made things worse).

Delaying the arrival ... (read more)

If Scott had used language like this, my guess is that the people he was trying to convince would have completely bounced off of his post.

I mostly agree with this, I wasn't suggesting he included that specific type of language, just that the arguments in the post don't go through from the perspective of most leader/highly-engaged EAs. Scott has discussed similar topics on ACT here but I agree the target audience was likely different.

I do think part of his target audience was probably EAs who he thinks are too critical of themselves, as I think he's written... (read more)

EDIT: Scott has admitted a mistake, which addresses some of my criticism:



(this comment has overlapping points with titotal's)

I've seen a lot of people strongly praising this article on Twitter and in the comments here but I find some of the arguments weak. Insofar as the goal of the post is to say that EA has done some really good things, I think the post is right. But I don't think it convincingly argues that EA has been net positive for the world.^[1]

First: based on surveys, it seems likely that most (not all!) highly-engaged/leader EAs believe GCRs/longt... (read more)

These were the 3 snippets I was most interested in

Under pure risk-neutrality, whether an existential risk intervention can reduce more than 1.5 basis points per billion dollars spent determines whether the existential risk intervention is an order of magnitude better than the Against Malaria Foundation (AMF). 

If you use welfare ranges that are close to Rethink Priorities’ estimates, then only the most implausible existential risk intervention is estimated to be an order of magnitude more cost-effective than cage-free campaigns and the hypothetical shr... (read more)

74 (= 0.074/0.01)

 .074/.01 is 7.4, not 74

Personally the FTX regrantor system felt like a nice middle ground between EA Funds and donor lotteries in terms of (de)centralization. I'd be excited to donate to something less centralized than EA Funds but more centralized than a donor lottery.

Which part of my comment did you find as underestimating how grievous SBF/Alameda/FTX's actions were? (I'm genuinely unsure)

Nitpick, but I found the sentence:

Based on things I've heard from various people around Nonlinear, Kat and Emerson have a recent track record of conducting Nonlinear in a way inconsistent with EA values [emphasis mine].

A bit strange in the context of the rest of the comment. If your characterization of Nonlinear is accurate, it would seem to be inconsistent with ~every plausible set of values and not just "EA values".

Appreciate the quick, cooperative response.

I want you to write a better post arguing for the same overall point if you agreed with the title, hopefully with more context than mine.

Not feeling up to it right now and not sure it needs a whole top-level post. My current take is something like (very roughly/quickly written):

1. New information is currently coming in very rapidly.
2. We should at least wait until the information comes in a bit slower before thinking seriously in-depth about proposed mitigations so we have a better picture of what went wrong. But "babbl

I thought I would like this post based on the title (I also recently decided to hold off for more information before seriously proposing solutions), but I disagree with much of the content.

A few examples:

It is uncertain whether SBF intentionally committed fraud, or just made a mistake, but people seem to be reacting as if the takeaway from this is that fraud is bad.

I think we can safely say with at this point >95% confidence that SBF basically committed fraud even if not technically in the legal sense (edit: but also seems likely to be fraud in the lega... (read more)

I’m not as sure about advisors, as I wrote here. Agree on recipients

It's a relevant point but I think we can reasonably expect EA leadership to do better at vetting megadonors than Sequoia due to (a) more context on the situation, e.g. EAs should have known more about SBF's past than Sequoia and/or could have found it out more easily via social and professional connections (b) more incentive to avoid downside risks, e.g. the SBF blowup matters a lot more for EA's reputation than Sequoia's.

To be clear, this does not apply to charities receiving money from FTXFF, that is a separate question from EA leadership.

Thanks for clarifying. To be clear, I didn't say I thought they were as bad as Leverage. I said "I have less trust in CEA's epistemics to necessarily be that much better than Leverage's , though I'm uncertain here"

I've read it. I'd guess we have similar views on Leverage, but different views on CEA. I think it's very easy for well-intentioned, generally reasonable people's epistemics to be corrupted via tribalism, motivated reasoning, etc.

But as I said above I'm unsure.

Edited to add: Either way, might be a distraction to debate this sort of thing further. I'd guess that we both agree in practice that the allegations should be taken seriously and investigated carefully, ideally by independent parties.

I agree that these can technically all be true at the same time, but I think the tone/vibe of comments is very important in addition to what they literally say, and the vibe of Arepo's comment was too tribalistic.

I'd also guess re: (3) that I have less trust in CEA's epistemics to necessarily be that much better than Leverage's , though I'm uncertain here (edited to add: tbc my best guess is it's better, but I'm not sure what my prior should be if there's a "he said / she said" situation, on who's telling the truth. My guess is closer to 50/50 than 95/5 in log odds at least).

I’m guessing I have a lower opinion of Leverage than you based on your tone, but +1 on Kerry being at CEA for 4 years making it more important to pay serious attention to what he has to say even if it ultimately doesn’t check out. We need to be very careful to minimize tribalism hurting our epistemics.

See also https://polymarket.com/market/will-binance-pull-out-of-their-ftx-deal and 

Does the "deliberate about current evidence" part includes thinking a lot about AI alignment to identify new arguments or considerations that other people on Earth may not have thought of, or would that count as new evidence?

It seems like if that would not count as new evidence, that the team you described might be able to come up with much better forecasts than we have today, and I'd think their final forecast would be more likely to end up much lower or much higher than e.g. your forecast. One consequence of this might then be be that your 90% confidence

Thanks Max! Was great seeing you as well. I did take some time off and was a bit more chill for a little while blogging however much I felt like. I've been doing a lot better for the past 2 months.

The impact isn't coming from the literal $10M donation to OpenAI, it's coming from spearheading its founding.

See https://twitter.com/esyudkowsky/status/1446562238848847877

“P(misalignment x-risk|AGI)”: Conditional on AGI being developed by 2070, humanity will go extinct or drastically curtail its future potential due to loss of control of AGI.

I'm guessing this definition is meant to separate misalignment from misuse, but I'm curious whether you are including either/both of these 2 cases as misalignment x-risk:

1. AGI is deployed and we get locked into a great outcome by today's standards, but we get a world with <=1% of the value of "humanity's potential". So we sort of have an existential catastrophe, without a discrete cata

In particular, I think many of the epistemically best EAs go into stuff like grant making, philosophy, general longtermist research, etc. which leaves a gap of really epistemically good people focusing full-time on AI. And I think the current epistemic situation in the AI alignment field (both technical and governance) is pretty bad in part due to this.

Thanks for clarifying. Might be worth making clear in the post (if it isn’t already, I may have missed something).

I mean EAs. I’m most confident about “talent-weighted EAs”. But probably also EAs in general.

(I've only skimmed the post)

This seems right theoretically, but I'm  worried that people will read this and think this consideration ~conclusively implies fewer people should go into AI alignment, when my current best guess is the opposite is true. I agree sometimes people make the argmax vs. softmax mistake and there are status issues, but I still think not enough people proportionally go into AI for various reasons (underestimating risk level, it being hard/intimidating, not liking rationalist/Bay vibes, etc.).

Thanks Will!

My dad just sent me a video of the Yom Kippur sermon this year (relevant portion starting roughly here) at the congregation I grew up in. It was inspired by longtermism and specifically your writing on it, which is pretty cool. This updates me emotionally toward your broad strategy here, though I'm not sure how much I should update rationally.

Ah, thanks for the pointer

Is the UAT mentioned anywhere in the bio anchors report as a reason for thinking DL will scale to TAI? I didn't find any mentions of it quickly ctrl-fing in any of the 4 parts or the appendices.

Yeah it's an EAF bug with crossposting linkposts from LessWrong. For now copy and paste the text into the browser and it will work, or click here.

• There was a vague tone of "the goal is to get accepted to EAG" instead of "the goal is to make the world better," which I felt a bit uneasy about when reading the post. EAGs are only useful in so far as they let community members to better work in the real world. 
  • Because of this, I don't feel strongly about the EAG team providing feedback to people on why they were rejected. The EAG team's goals isn't to advise on how applicants can fill up their "EA resume." It's to facilitate impactful work in the world. 
• I remembered a comment that I really lik

Good idea, I made a very quick version. Anyone should feel free to update it.

Yeah I realized this when proofreading and left it as I thought it drove home my point well :p

I think a lot of the value of university is providing a peer group for social and intellectual development. To the extent you hasn't found a great group of friends at university, I think you should actively try very hard to find a group that you enjoy spending time with then lean into this. To the extent you can fill this need outside of university, it seems very reasonable to drop out.

This is the advice I'd give to my younger self: I didn't make many great friends in college and should have tried much harder to find my niche. But I don't think I was ready... (read more)

FYI: You can view community median forecasts for each question at this link. Currently it looks like:

Thanks for the suggestion and glad you found the forecasts helpful :)

I personally have a distaste for academic credentialist culture so am probably not the best person to turn this into a more prestigious looking report. I agree it might be valuable despite my distaste, so if anyone reading this is interested in doing so feel free to DM me and I can probably help with funding and review if you have a good writing track record.

It sounds like at least part of your argument could be summarized as: "Will MacAskill underrates x-risk relative to most longtermists; The Precipice is a better introduction because it focuses on x-risk."

I don't have a strong view about the focus on x-risk in general. I care most about the lack of clarity on which areas are highest priority, and what I believe is a mistake in not focusing on AI enough (and the wrong emphasis within AI).

In the post I wrote:

One big difference between The Precipice and WWOTF that I’m not as sure about is the framing of reduci

I do think it's more about whether you're doing things in such a way that if they knew why you were doing them, they'd mostly not be bothered (ie passing the red face test). But that doesn't really solve the problem that digital sentience is a weird reason to do a lot of things, and there are lots of things I endorse it being inappropriate to be too explicit about.

Of course this is a spectrum, and we shouldn't put up a public website listing all our beliefs including the most controversial ones or something like that (no one in EA is very close to this ext... (read more)

I think steps 1 and 2 in your chain are also questionable, not just 3-5.

1. Want to maximize number of EAs 

Why do we want to maximize number of EAs, this seems very non-obvious to me? Some people would add much more to the community than others via epistemics, culture, direct talent, etc. If we added enough of certain types of people to the community, especially too quickly, it could easily be net negative.

2. Use framings, arguments and examples that you don't think hold water but work at getting people to join your group [I don't think EAs do this, I'm g

I didn't mean to imply that. I think we very likely need to solve alignment at some point to avoid existential catastrophe (since we need aligned powerful AIs to help us achieve our potential), but I'm not confident that the first misaligned AGI would be enough to cause this level of catastrophe (especially for relatively weak definitions of "AGI").