All of Arepo's Comments + Replies

I'm still highly sceptical of neglectedness as anything but a first-pass heuristic for how prioritisation organisations might spend their early research - firstly because there are so many ways a field can be 'not neglected' and still highly leveraged (e.g. Givewell and Giving What We Can were only able to have the comparative impact they did because the global health field had been vigorously researched but no-one had systemically done the individual-level prioritisation they did with the research results); secondly because it encourages EA to reject esta... (read more)

Sure, so we agree?

Ah, sorry, I misunderstood that as criticism.

Do you think that forecasting like this will hurt the information landscape on average? 

I'm a big fan of the development e.g. QRI's process of making tools that make it increasingly easy to translate natural thoughts into more usable forms. In my dream world, if you told me your beliefs it would be in the form of a set of distributions that I could run a monte carlo sim on, having potentially substituted my own opinions if I felt differently confident than you (and maybe beyond that there'... (read more)

I weakly disagree here. I am very much in the "make up statistics and be clear about that" camp.

I'm sympathetic to that camp, but I think it has major epistemic issues that largely go unaddressed:

• It systemically biases away from extreme probabilities (it's hard to assert < than ${10}^{-3}$, for e.g., but many real-world probabilities are and post-hoc credences look like they should have been below this)
• By focusing on very specific pathways towards some outcome, it diverts attention towards easily definable issues, and hence away from the prospe

Confirmed, there's still people around :)

I would strongly push back on the idea that a world where it's unlikely and we can't change that is uninteresting. In that world, all the other possible global catastrophic risks become far more salient as potential flourishing-defeaters.

Thanks for the shout-out :) If you mentally replace the 'multiplanetary' state with 'post-AGI' in this calculator, I do think it models the set of concerns Will's talking about here pretty well.

Thanks Ozzie! I'll definitely try this out if I ever finish my current WIP :)

Questions that come to mind:

• Will it automatically improve as new versions of the underlying model families are released?
• Will you be actively developing it?
• Feature suggestion: could/would you add a check for obviously relevant literature and 'has anyone made basically this argument before'?

A 10% chance of transformative AI this decade justifies current EA efforts to make AI go well.

Not necessarily. It depends on 

a) your credence distribution of TAI after this decade, 

b) your estimate of annual risk per year of other catastrophes, and 

c) your estimate of the comparative longterm cost of other catastrophes.

I don't think it's unreasonable to think, for example, that

• there's a very long tail to when TAI might arrive, given that its prospects of arriving in 2-3 decades are substantially related to to its prospects of arriving this d

But the expected value of existential risk reduction is—if not infinite, which I think it clearly is in expectation—extremely massive.

I commented something similar on your blog, but as soon as you allow that one decision is infinite in expectation you have to allow that all outcomes are, since whatever possibility of infinite value you have given that action must still be present without it.

If you think the Bostrom number of 10^52 happy people has a .01% chance of being right, then you’ll get 10^48 expected future people if we don’t go extinct, meaning red

I agree with Yarrow's anti-'truth-seeking' sentiment here. That phrase seems to primarily serve as an epistemic deflection device indicating 'someone whose views I don't want to take seriously and don't want to justify not taking seriously'.

I agree we shouldn't defer to the CEO of PETA, but CEOs aren't - often by their own admission - subject matter experts so much as people who can move stuff forwards. In my book the set of actual experts is certainly murky, but includes academics, researchers, sometimes forecasters, sometimes technical workers - sometime... (read more)

I agree that the OP is too confident/strongly worded, but IMO this

which is more than enough to justify EA efforts here. 

could be dangerously wrong. As long as AI safety consumes resources that might have counterfactually gone to e.g. nuclear disarmament, stronger international relations, it might well be harmful in expectation.

This is doubly true for warlike AI 'safety' strategies like Aschenbrenner's call to intentionally arms race China, Hendrycks, Schmidt and Wang's call to 'sabotage' countries that cross some ill-defined threshold, and Yudkowsky c... (read more)

Well handled, Peter! I'm curious how much of that conversation was organic, how much scripted or at least telegraphed in advance?

I'm warming to CoGi

That makes some sense, but leaves me with questions like

• Which projects were home runs, and how did you tell that a) they were successful at achieving their goals and b) that their goals were valuable?
• Which projects were failures that you feel were justifiable given your knowledge state at the time?
• What do these past projects demonstrate about the team's competence to work on future projects?
• What and how was the budget allocated to these projects, and do you expect future projects projects to have structurally similar budgets?
• Are there any other analogies y

IMO it would help to see a concrete list of MIRI's outputs and budget for the last several years. My understanding is that MIRI has intentionally withheld most of its work from the public eye for fear of infohazards, which might be reasonable for soliciting funding from large private donors but seems like a poor strategy for raising substantial public money, both prudentially and epistemically. 

If there are particular projects you think are too dangerous to describe, it would still help to give a sense of what the others were, a cost breakdown for tho... (read more)

Might be confusing with SoGive.

Should our EA residential program prioritize structured programming or open-ended residencies?

You can always host structured programs, perhaps on a regular cycle, but doing so to the exclusion of open-ended residencies seems to be giving up much of the counterfactual value the hotel provided. It seems like a strong overcommitment to a concern about AI doom in the next low-single-digit years, which remains (rightly IMO) a niche belief even in the EA world, despite heavy selection within the community for it.

Having said that, to some degree it sounds l... (read more)

Thanks for the extensive reply! Thoughts in order:

I would also note that #3 could be much worse than #2 if #3 entails spreading wild animal suffering.

I think this is fair, though if we're not fixing that issue then it seems problematic for any pro- longtermism view, since it implies the ideal outcome is probably destroying the biosphere. Fwiw I also find it hard to imagine humans populating the universe with anything resembling 'wild animals', given the level of control we'd have in such scenarios, and our incentives to exert it. That's not to say we could... (read more)

Very helpful, thanks! A couple of thoughts:

• EA grantmaking appears on a steady downward trend since 2022 / FTX.

It looks like this is driven entirely by Givewell/global health and development reduction, and that actually the other fields have been stable or even expanding.

Also, in an ideal world we'd see funding from Longview and Founders Pledge. I also gather there's a new influx of money into the effective animal welfare space from some other funder, though I don't know their name.

Kudos to whoever wrote these summaries. They give a great sense of the contents and at least wth mine capture the essence of it much more succinctly than I could!

Thanks! The online courses page describes itself as a collection of 'some of the best courses'. Could you say more about what made you pick these? There are dozens or hundreds of online courses these days (esp on general subjects like data analysis), so the challenge for pursuing them is often a matter of filtering convincingly.

Good luck with this! One minor irritation with the structure of the post is I had to read halfway down to find out which 'nation' it referred to. Suggest editing the title to 'US-wide', so people can see at a glance if it's relevant to them.

I remember him discussing person-affecting views in Reasons and Person, but IIRC (though it's been a very long time since I read it) he doesn't particularly advocate for them. I use the phrase mainly because of the quoted passage, which appears (again IIRC) in both The Precipice and What We Owe the Future, as well as possibly some of Bostrom's earlier writing. 

I think you could equally give Bostrom the title though, for writing to my knowledge the first whole paper on the subject.

Cool to see someone trying to think objectively about this. Inspired by this post, I had a quick look at the scores on the world happiness report to compare China to its ethnic cousins, and while there are many reasons to take this with a grain of salt, China does... ok. On 'life evaluation', which appears to be the all things considered metric (I didn't read the methodology, correct me if I'm wrong), some key scores:

Taiwan: 6.669

Philippines: 6.107

South Korea: 6.038

Malaysia: 5.955

China: 5.921

Mongolia: 5.833

Indonesia: 5.617

Overall it's ranked 68th of 147 li... (read more)

Thanks :)

I don't think I covered any specific relationship between factors in that essay (except those that were formally modelled in), where I was mainly trying to lay out a framework that would even allow you to ask a question. This essay is the first time I've spent meaningful effort on trying to answer it.

I think it's probably ok to treat the factors as a priori independent, since ultimately you have to run with your own priors. And for the sake of informing prioritisation decisions, you can decide case by case how much you imagine your counterfactual action changing each factor.

You don't need a very high credence in e.g. AI x risk for it to be the most likely reason you and your family die

I think this is misleading, especially if you agree with the classic notion of x-risk as excluding events from which recovery is possible. My distribution of credence over event fatality rates is heavily left-skewed, so I would expect far more deaths under the curve between 10% and 99% fatality than between 99% and 100%, and probably more area to the left even under a substantially more even partition of outcomes. 

I fear we have yet to truly refute Robin Hanson’s claim that EA is primarily a youth movement.

FWIW my impression is that CEA have spent significantly more effort on recruiting people from universities than any other comparable subset of the population.

Somehow despite 'Goodharting' being a now standard phrase, 'Badharting' is completely unrecognised by Google. 

I suggest the following intuitive meaning: failing to reward a desired achievement, because the proxy measure you used to represent it wasn't satisfied:

'No bonus for the staff this year: we didn't reach our 10% sales units growth target.'

'But we raised profits by 30% by selling more expensive products, you Badharting assholes!'

Comically, the job ad is paywalled.

I guess in general any decision binds all future people in your lightcone to some counterfactual set of consequences. But it still seems practically useful in interpersonal interactions to distinction a) between those that deliberately restrict their action set/those that just provide them in expectation with a different access set of ~the same size, and b) between those motivated by indifference/those motivated specifically by an authoritarian desire to make their values more consistently with ours.

Muchos hugs for this one. I'm selfishly glad you were in London long enough for us to meet, fwiw :)

I feel like this is a specific case of a general attitude in EA that we want to lock in our future selves to some path in case our values change. The more I think about this the worse it feels to me, since 

a) your future values might in fact be better than your current ones, or if you completely reject any betterness relation between values then it doesn't matter either way

b) your future self is a separate person. If we imagine the argument targeting any... (read more)

I'm not doing the course, but I'm pretty much always on the EA Gather, and usually on for coworking and accountabilitying compatible with my timezone (UTC+8). Feel free to hop on there and ping me - there's a good chance I'll be able to reply at least by text immediately, and if not pretty much always within 12 hours.

To be strictly accurate, perhaps I should have said 'the more you know about AI risks and AI safety, the higher your p(doom)'. I do think that's an empirically defensible claim. Especially insofar as most of the billions of people who know nothing about AI risks have a p(doom) of zero.

That makes it sound like a continuous function when it isn't really. Sure, people who've never or barely thought about it and then proceed to do so are likely to become more concerned - since they have a base of ~0 concern. That doesn't mean the effect will have the same shap... (read more)

I'm really glad you gave this talk, even as something of a sceptic of AI x-risk. As you say, this shouldn't be a partisan issue. I would contest one claim though:

> Generally, the more you know about AI, the higher your p(doom), or estimated probability that ASI would doom humanity to imminent extinction.

I don't see the evidence for this claim, and I keep seeing people in the doomer community uncritically repeat it. AI wouldn't be progressing if everyone who understood it became convinced it would kill us all. 

Ok, one might explain this progress via... (read more)

I resonate strongly with this, to the extent that I worry most of the main EA 'community-building' events don't actually do anything to build community. I can count on the fingers of 0 hands the number of meaningful lasting relationships I've subsequently developed with anyone I met at an EAG(x), after having been to maybe 10 over the course of 8ish years. 

That's not to say there's not value in the shorter term relationships that emerge from them - but by comparison I still think fondly of everyone I met at a single low-cost EA weekend retreat over a ... (read more)

Good luck with this - I've always wished for more global EA events!

To clarify, is this explicitly a rebranding of EAGx Virtual, or is it meant to be something qualitatively different? If the latter, could you say more about what the differences will be?

These were all (with the exception of 3) written in the six months following FTX, when it was still being actively discussed, and at least three other EA-relateed controversies had come out in succession. So I probably misremembered FTX specifically (thinking again, IIRC there was an FTX tag that was the original 'autodeprioritised' tag, whose logic maybe got reverted when the community separation logic was introduced). But I think it's fair to say that these were a de facto controversy-deprioritisation strategy (as the fourth post suggests) rather than a measured strategy that could be expected to stay equally relevant in calmer times.

I feel like people are treating the counterfactual as 'no way to filter out community posts'; whereas the forum software currently allows you to filter for any given tag, and could easily be tweaked to (or possibly already allows you to) filter out a particular tag.

So the primary counterfactual isn't 'no separation' it's 'greater transparency and/or community involvement in what gets tagged "community"'.

You may well be right that CEA is biased (its hard not to be) and the criteria could be 
made clearer.

My suggestion if the current separation is kept would be to reallow community tagging of posts, but require it to go above a certain threshold and/or have a delay, so that posts don't bounce back and forward between the two feeds.

I'm also not sure community posts get less attention (forum team can tell me).

FWIW I suspect both that being tagged community causes reduced attention, but that they get less attention overall since many low-karma posts slide ... (read more)

I would love to see more events like this in the community. Honestly, those handful of low-budget attendee-run retreats I've been to have been worth substantially more than EAG(x)s 

Increase relative to what counterfactual? I think it might be true both that annual risk of bad event goes up from AI, and that all-time risk decreases (on the assumption that we're basically reaching a hurdle we have to pass anyway, and that I'm highly sceptical that we gain much in practice by implementing forceful procedures that slow us down from getting there).

Weakly in favour of moderate regulation, though I think within the EA movement the extinction case is much overstated, the potential benefits are understated, and I can imagine regulatory efforts backfiring in any number of ways.

Having said that after I voted I realised there are a number of conditional actions lower down in the poll that I would be sympathetic to given the conditions (e.g. pause if mass unemployment or major disaster)

You might be right re forecasting (though someone willing in general to frequently bet on 2% scenarios manifesting should fairly quickly outperform someone who frequently bets against them - if their credences are actually more accurate). 

I think you’re wrong about UK AISI not putting much credence on extinction scenarios? I’ve seen job adverts from AISI talking about loss of control risk (i.e., AI takeover), and how ‘the risks from AI are not sci-fi, they are urgent.’ And I know people working at AISI who, last I spoke to them, put ≫10% on

In the supplement, I assume (for simplicity) that we just can't affect long-term existential risk.

But I also think that, if there are ways to affect it, that'll seem more like interventions to increase Flourishing than to increase Survival. (E.g. making human decision-making wiser, more competent, and more reflective).

IMO the mathematical argument for spreading out to other planets and eventually stars is a far stronger a source of existential security than increasing hard-to-pin-down properties like 'wisdom'.

If different settlements' survival were indepen... (read more)

because it's a very complex state of arguments, the vast majority of the world misunderstands it pretty severely... They have very clearly missed some important points of argument.  

This seems like an argument from your own authority. I've read a number of doomer arguments and personally found them unconvincing, but I'm not asking anyone to take my word for it. Of course you can always say 'you've read the wrong arguments', but in general, if your argument amounts to 'you need to read this 10s-of-thousands-of-words-argument' there's no reason for an o... (read more)

Here's the problem: The more people think seriously about this question, the more pessimistic they are. 

Citation needed on this point. I you're underrepresenting the selection bias for a start - it's extremely hard to know how many people have engaged with and rejected the doomer ideas since they have far less incentive to promote their views. And those who do often find sloppy argument and gross misuses of the data in some of the prominent doomer arguments. (I didn't have to look too deeply to realise the orthogonality thesis was a substantial source... (read more)

I agree with the general thrust of this post - at least the weak version that we should consider this to be an unexplored field, worth putting some effort into. But I strong disagree with the sentiment here:

maybe we’re closer to the “ceiling” on Survival than we are to the “ceiling” of Flourishing.

Most people (though not everyone) thinks we’re much more likely than not to Survive this century.  Metaculus puts *extinction* risk at about 4%; a survey of superforecasters put it at 1%. Toby Ord put total existential risk this century at 16%.

Not ... (read more)

Yeah, it's a relatively objective expression of the ratio of how much the most successful people in society care about tiny benefits to themselves vs any alternative.