All of Arepo's Comments + Replies

That's sad. For anyone interested in why they shut down (I'd thought they had an indefinitely sustainable endowment!), the archived version of their website gives some info:

Over time FHI faced increasing administrative headwinds within the Faculty of Philosophy (the Institute’s organizational home).  Starting in 2020, the Faculty imposed a freeze on fundraising and hiring.  In late 2023, the Faculty of Philosophy decided that the contracts of the remaining FHI staff would not be renewed.  On 16 April 2024, the Institute was closed down.

By inference, if you are one of those copies, the 'moral worth' of your own perceived torture will therefore be 1/10billionth of its normal level. So, selfishly, that's a huge upside - I might selfishly prefer being one of 10 billion identical torturees as long as I uniquely get a nice back scratch afterwards, for e.g.

Space lasers don't seem as much of a threat as Jordan posits. They have to be fired from somewhere. If that's within the solar system they're targeting, then that system will still have plenty of time to see the object that's going to shoot them arriving. If they're much further out, it becomes much harder both to aim them correctly and to provide enough power to keep them focused, and the source needs to be commensurately more powerful (as in more expensive to run), and with a bigger lens, so more visible while under constructive and more vulnerable to co... (read more)

I don't think anyone's arguing current technology would allow self-sufficiency. But part of the case for offworld settlements is that they very strongly incentivise technolology that would.

In the medium term, an offworld colony doesn't have to be fully independent to afford a decent amount of security. If it can a) outlast some globally local catastrophe (e.g. a nuclear winter or airborne pandemic) and b) get back to Earth once things are safer, it still makes your civilisation more robust. 

I broadly agree with the arguments here. I also think space settlement has a robustness to its security that no other defence against GCRs does - it's trivially harder to kill all of more people spread more widely than it is to kill of a handful on a single planet. Compare this to technologies designed to regulate a single atmosphere to protect against biorisk, AI safety mechanisms that operate on AGIs whose ultimate nature we still know very little of, global political institutions that could be subverted or overthrown, bunkers on a single planet, etc, al... (read more)

I strongly agree with the first half of this post - bunkers and refuges are pretty bad as a defence against global catastrophes.

Your solution makes a lot less sense to me. It seems like it has many of the same problems you're trying to avoid - it won't be pressure tested until the world collapses. In particular, if it's an active part of a local community, that implies people will be leaving and reentering regularly, which means any virus with a long incubation period could be in there before people know it's a problem. 

Also, I feel like your whole li... (read more)

Hey Corentin,

The calculators are intentionally silent on the welfare side, on the thought that in practice it's much easier to treat as a mostly independent question. That's not to say it actually is independent, and ideally I would like the output to include more information about what the pathways to either extinction or an interstellar state, so that people can do some further function on the output. I do think it's reasonable, even on a totalising view, to prioritise improving future welfare conditional on it existing and largely ignoring the question ... (read more)

Yeah, it sounds like this might not be appropriate for someone with your credences, though I'm confused by what you say here:

I mentioned point/mean probability estimates, but my upper bounds (e.g. 90th percentile) are quite close, as they are strongly limited by the means. For example, if one's mean probability is 10^-10, the 90th percentile probability cannot be higher than 10^-9, otherwise the mean probability would be higher than 10^-10 (= (1 - 0.90)*10^-9), which is the mean. So my point remains as long as you think my point/mean estimates are reasonab

Hm, the link works ok for me. What happens when you open it? It can be a bit shonky on mobile phones - maybe try using it on a laptop/desktop if you haven't.

It's called 'EA coworking and lounge', if that helps.

Thanks for the kind words, David. And apologies - I'd forgotten you'd published those explicit estimates. I'll edit them in to the OP.

My memory of WWOtF is that Will talks about the process, but other than giving a quick estimate of '90% chance we recover without coal, 95% chance with' he doesn't do as much quantifying as you and Luisa. 

Also Lewis Dartnell talked about the process extensively in The Knowledge, but I don't think he gives any estimate at all about probabilities (the closest I could find was in an essay for Aeon where he opined that 'an industrial revolution without coal would be, at a minimum, very difficult').

Hey Vasco, thanks for the in-depth reply, and thanks again for trawling over this behemoth :)

Let me take these points in order:

I think the annual risk of human extinction not involving transformative AI (TAI) is astronomically low.

I'm highly sceptical of point probability estimates for events for which we have virtually no information - that's exactly why I made these tools. Per Dan Schwarz's recent post, it seems much more important to me to give an interactive model into which people can put their own credences, so that we can then debate the input ... (read more)

I'm happy to talk you through using it if you're finding it confusing.

If you (or anyone else) reading this wants to catch me for some support, I'm on the EA Gather Town as much as possible (albeit currently in New Zealand time), so you can log in there and ping me :)

I think it would have been better to speak up way, way sooner,

If and when this postmortem ever does happen, I hope they will address this, too. The lack of public engagement on the subject with the rest of the movement following the FTX disaster seems a comparable lapse of responsibility to anything that might have happened in the time leading up to it.

Rebecca's comments seem consistent with Beckstead being part of her concern, though.

I found this interesting, and a model I've recently been working on might be relevant - I've emailed you about it. One bit of feedback:
 

Please reach out to hello@futuresearch.ai if you want to get involved!

You might want to make it more clear what kind of collaboration you're hoping to receive.

I think you gave up on your theory being maximally consistent when you opted for diversity of experience as a metavalue. Most people don't actually consider their own positive experiences cheapened by someone on the other side of the world having a similar experience.

Also, if you're doing morality by intuition (a methodology I think has no future), then I suspect most people would much sooner drop 'diversity of experience good' than 'torture bad'.

This. I'm imagine some Abrodolph Lincoler-esque character - Abronard Willter maybe - putting me in a brazen bull and cooing 'Don't worry, this will all be over soon. I'm going to create 10billion more of you also in a brazen bull, so the fact that I continue to torture you personally will barely matter.'

most people intrinsically value diversity of experience, and see a large number of very similar lives as less of a good thing.

Especially in such a contentious argument, I think it's bad epistemics to link to a page with some random dude saying he personally believes x (and giving no argument for it) with the linktext 'most people believe x'.

I hadn't even thought of that! Yeah, that's some pretty impressive hypocrisy.

The Copenhagen interpretation of ethics strikes again.

This doesn’t seem so different from p-zombies, and probably some moral thought experiments.

I'm not sure what you mean here. That the simulation argument doesn't seem different from those? Or that the argument that 'we have no evidence of their existence and therefore shouldn't update on speculation about them' is comparable to what I'm saying about the simulation hypothesis? 

If the latter, fwiw, I feel the same way about p-zombies and (other) thought experiments. They are a terrible methodology for reasoning about anything, very occasionally the only ... (read more)

I think assuming that this is purely based on optics is unwarranted. Like I argued at the time, talk of 'optics' is kind of insulting to the everyperson, carrying the implication that the irrational public will misunderstand the +EV of such a decision. Whereas I contend that there's a perfectly rational Bayesian update that people should do towards an organisation being poorly run or even corrupt when that org spends large sums of money on vanity projects which they justify with a vague claim about having done some CBA that they don't want to share.

Meanwhi... (read more)

1. We may ourselves be simulated in a similar way without knowing it, if our entire reality is also simulated. We wouldn't necessarily have access to what the simulation is run on.

It seems weird to meaningfully update in favour of some concrete view on the basis that something might be true but that

1. we have no evidence for it, and 
2. if it is true then everything we know about the universe is equally undermined

Is there are online version of the case for the fading qualia argument? This feels a bit abstract without it...

Partly from a scepticism about the highly speculative arguments for 'direct' longtermist work - on which I think my prior is substantially lower than most of the longtermist community (though I strongly suspect selection effects, and that this scepticism would be relatively broadly shared further from the core of the movement).

Partly from something harder to pin down, that good outcomes do tend to cluster in a way that e.g. Givewell seem to recognise, but AFAIK have never really tried to account for (in late 2022, they were still citing that post while say... (read more)

Hey Johannes :)

To be clear, I think the original post is uncontroversially right that it's very unlikely that the best intervention for A is also the best intervention for B. My claim is that, when something is well evidenced to be optimal for A and perhaps well evidenced to be high tier for B, you should have a relatively high prior that it's going to be high tier or even optimal for some related concern C.

Where you have actual evidence available for how effective various interventions are for C, this prior is largely irrelevant - you look at the evidence... (read more)

This statement was very surprising to me:
 

The “concerned” participants (all of whom were domain experts) ... the “skeptical” group (mainly “superforecasters”) 

Can you say more about your selection process, because this seems very important to understanding how much to update on this. Did you 

a) decide you needed roughly equally balanced groups of sceptics vs concerned, start with superforecasters, find that they were overwhelmingly sceptics, and therefore specifically seek domain experts because they were concerned

b) decide you needed roughl... (read more)

Interesting stuff. I'm sceptical a priori, but it would be amazing if this kind of thing replicated. I think there's a typo:

… and four to six months after treatment, they…

In contrast, after...

I think it would be a little bit of a surprising and suspicious convergence if the best interventions to improve human health (e.g. GiveWell's top charities) were also the best to reliably improve global capacity

Fwiw, I think Greg's essay is one of the most overweighted in forum history (as in, not necessarily overrated, but people put way too much weight in its argument). It's a highly speculative argument with no real-world grounding, and in practice we know that of many well-evidenced socially beneficial causes that do seem convergently beneficial in ot... (read more)

greater confidence in EEV lends itself to supporting longshots to reduce x-risk or otherwise seek to improve the long-term future in a highly targeted, deliberate way.

This just depends on what you think those EEVs are. Long-serving EAs tend to lean towards thinking that targeted efforts towards the far future have higher payoff, but that also has a strong selection effect. I know many smart people with totalising consequentialist sympathies who are sceptical enough of the far future that they prefer to donate to GHD causes. None of them are at all active in the EA movement, and I don't think that's coincidence.

I'm in New Zealand atm, so this is uncomfortably early for me, but I'll try and make it!

I think much of this criticism is off. There are things I would disagree with Nuno on, but most of what you're highlighting doesn't seem to fairly represent his actual concerns.

Nuño never argues for why the comments they link to shouldn't be moderated

He does. Also, I suspect his main concern is with people being banned rather than having their posts moderated.

Nuňo doesn't make points that EA is too naïve/consequentialist/ignoring common-sense enough. Instead, they don't think we've gone far enough into that direction. See in Alternate Visions of EA, the cl

Cool! I just submitted a project - minor bit of feedback is that it's slightly irritating to have the 'project subtitle' field be mandatory.

Great post - I'm embarrassed to have missed it til now! One key point I disagree with:

there might be interventions that reduce risk a lot for not very long or not very much but for a long time. But actions that drastically reduce risk and do so for a long time are rare.

I think there are two big possible exceptions to the latter claim: benign AI and becoming sustainably multiplanetary. EAs have discussed the former a lot, and I don't have much to add (though I'm highly sceptical of it as an arbitrary-value lock-in mechanism on cosmic timelines). I think the... (read more)

Katja and I date, so yes, I am biased, but I really think that’s a pretty unimportant fact about her

Congrats to both of you on your great catches! Say hi to her for me - it's been a while :)

More generally, what incentives exist? In a normal for-profit environment there are various reasons for individuals to start their own company, to seek promotion, to do a good job, to do a bad job, to commit institutional fraud etc - we typically think of these as mainly financial, and often use the adage 'follow the money' as a methodology to try and discover these phenomena, to encourage the good ones and discourage the bad. 

I want to know what the equivalent methodology would be to find out equivalent phenomena at EA organisations.

EA organizations don't really have a great need for nurses, for history professors, for plumbers, etc.

Fwiw, I was involved with an EA organisation that  that struggled for years with the admin of finding trustworthy tradespeople (especially plumbers).

More generally, I think a lot of EA individuals would benefit a lot from access to specialist knowledge from all sorts of fields, if people with that knowledge were willing to offer it free or at a discount to others in the community. 

I have a stronger version of the same concerns, fwiw. I can't imagine a 'Long Reflection' that didn't involve an extremely repressive government clamping down on private industry every time a company tried to do anything too ambitious, and that didn't effectively promote some caste of philosopher kings above all others to the resentment of the populace. It's hard to believe this could lead to anything other than substantially worse social values.

I also don't see any a priori reason to think 'reflecting' gravitates people towards moral truth or better values. Philosophers have been reflecting for centuries, and there's still very little consensus among them or any particular sign that they're approaching one.