All of acylhalide's Comments + Replies

Why AGI Timeline Research/Discourse Might Be Overrated

Thanks for replying!

Rest of my comment is conditional on EA folks having consensus for >50% by 2040. I personally don't actually believe this yet, but let's imagine a world where you and I did. This discussion would still be useful if we are one day year X and we knew >50% odds by year X + 20.

And there are other causes where we can have a positive effect without directly competing for resources

I do feel it competes for resources though. Attention in general is a finite resource. 

 - People learning of EA have a finite attention span and can ... (read more)

Why AGI Timeline Research/Discourse Might Be Overrated

Single data point but I think my life trajectory could look different if I believed >50% odds of AGI by 2040, versus >50% odds by 2100. And I can't be the only one who feels this way.

On the 2100 timeline I can imagine trusting other people to do a lot of the necessary groundwork to build the field globally till it reaches peak size. On the 2040 timeline every single month matters and it's not obvious the field will grow to peak size at all. So I'd feel much more compelled to do field building myself. And so would others even if they have poor persona... (read more)

I tend to think diversification in EA is important even if we think there's a high chance of AGI by 2040. Working on other issues gives us better engagement with policy makers and the public, improves the credibility of the movement, and provides more opportunities to get feedback on what does or doesn't work for maximizing impact. Becoming insular or obsessive about AI would be alienating to many potential allies and make it harder to support good epistemic norms. And there are other causes where we can have a positive effect without directly competing for resources, because not all participants and funders are willing or able to work on AI.
Things that limit my ambition in EA

These are real factors and major reasons people are not ambitious.

Some level of financial stability is prudent. Beyond that you just have to  care about the goal more than securing complete stability. Lots of people don't stay at early-stage startups because they can get good enough pay at later stage ones.

Burnout is real and there are tips you will find (on this site as well as elsewhere). Many people don't reduce this odds of burnout to zero (I don't know if they even can), but you can reduce it by some amount.

Imposter syndrome and aversion to rejec... (read more)

This is very good advice and I think I need to change the mindset and focus more on the goal. That's a great observation that I hadn't heard before. Really with networking being so important it makes sense that one would have to go the extra mile to effectively manage rejection.
On Deference and Yudkowsky's AI Risk Estimates

Not sure why this is on EAF rather than LW or maybe AF, but anyway

One obvious answer is LW community and mods tend to defer to yudkowksy more than EAF connunity.

(This doesn't argue whether the deferrence is good or bad, but this difference is a fact about reality I think)

What are EA's biggest legible achievements in x-risk?

Thanks for replying. I'm not sure this satisfies the criteria of "legible" as I was imagining it, since I buy most of the AI risk arguments and still feel poorly equipped to evaluate how important this was. But I do not have sufficient ML knowledge; perhaps it was legible to people with sufficient knowledge.

P.S. If it doesn't take too much of your time I would love to know if there's any discussion on why this was significant for x-risk, say, on the alignment forum. I found the paper and OpenAI blogpost but couldn't find discussion. (If it will take time I totally understand, I will try finding it myself.)

4Rohin Shah16d
I don't think there's great public discussion of that.
A Critical Review of Open Philanthropy’s Bet On Criminal Justice Reform

Thanks for the reply. This makes a lot of sense! I will look into Bridgewater.

A Critical Review of Open Philanthropy’s Bet On Criminal Justice Reform

Oh okay I see. Thank you for responding!

You have shifted my opinion a bit, it's no longer obvious to me where exactly the line between private and public information should be drawn here.

I can see why releasing information about personal incompetence for instance might be unusual in some cultures; I'm not sure why you can't build a culture where releasing such information is accepted.

You're right OpenPhil not being a grant recommendation service changes things a bit.

I think I would still be keen on seeing more research (or posts or comments etc) that tries... (read more)

Good to hear it shifted your opinion!

> I can see why releasing information about personal incompetence for instance might be unusual in some cultures; I'm not sure why you can't build a culture where releasing such information is accepted.

I agree it's possible, but think it's a ton of work! Intense cultural change is really tough. 

Imagine an environment, for instance, where we had a public ledger of, for every single person:

  • How much good and bad they've done.
  • Their personal strengths and weaknesses.
  • Their medical and personal issues. 

There would... (read more)

A Critical Review of Open Philanthropy’s Bet On Criminal Justice Reform

However, it would likely be very awkward and unprofessional to actually release this information publicly.

IMO this is a norm that should urgently change. AFAIK 80k hours and CEA have admitted mistakes and changes in cause prioritisation before, including the whole global health to longtermist shift.

Willingness to admit mistakes is table stakes for good epistemics.

In this case, the "mistakes" is often a list of things like, "This specific organization was much worse than we thought.  The founders happened to have issues A, B, and C, which really hurt the organization's performance". 

Releasing such information publicly is a big pain and something our culture is not very well attuned to. If OP basically announced information like, "This small group we funded is terrible, in large part because their CEO, George, is very incompetent", that would be very unusual and there would likely be a large amount of resis... (read more)

A Critical Review of Open Philanthropy’s Bet On Criminal Justice Reform

Interesting link, thanks for sharing! I can think of counterpoints to that assumption too but won't discuss here as its offtopic. (I mean I agree it is a real effect, its strength can vary I think.)

Are too many young, highly-engaged longtermist EAs doing movement-building?

As an undergrad who has considered doing movement-building despite probably having more fit for direct research, you're basically right.

The biggest reason for not doing direct research though is shortage of mentorship. Expecting someone to write their own research agendas, apply for funding and work solo straight after undergrad is a tall order. There is no clear feedback loop and there is less clearly visible scope for upskilling, which as someone junior you intrinsically tend to consider valuable. It is easier to take the lower risk path which is apply f... (read more)

A Critical Review of Open Philanthropy’s Bet On Criminal Justice Reform

I see. That statement of yours brought to my mind questions like:

 - can we assume (most) criminals are rational actors? is there evidence?

 - how do you compare say, years in prison versus benefits from getting away with a theft in an EV calculation?

 - how do you consider personal circumstances? (a poor thief gets more utility than a rich thief)

 - does the punitive justice system currently do any calculations of this sort?


I don't think any of this discussion is essential to the piece, it's just that that line caught my eye as smuggling in a whole perspective.

So I don't actually have to assume that criminals are rational actors; I can also assume that actions which are high expected value will spread through mimesis. See this short post: Unconscious Economics [] for an elaboration of this point. But you are right that it smuggles many assumptions.
What is the right ratio between mentorship and direct work for senior EAs?

Not a complete answer, but some subquestions this question invokes for me:

 - How much difference does lack of mentorship have on junior researchers? Are they likely to quit EA research altogether and do something else, are they likely to take more years to reach same capability level, are they likely to simply fail to reach the same level of capability despite trying?

 - Split above question by more minimal mentorship and deeper mentorship. I have a feeling deeper mentoring relationships can be very valuable (I could be wrong).

 - Mentorship h... (read more)

Steering AI to care for animals, and soon

If digital minds have moral status comparable to biological ones, will this matter as much?

a) Digital minds are more matter- and energy-efficient, so we can have much more of them than biological minds.

b) If we deploy transformative AI that is aligned, we likely can eventually get digital minds if we want. (Atleast under naturalist worldviews)

c) Digital minds optimised for any metric probably do not look identical to either human or animal minds. Examples of metrics include any function of capability, energy-efficiency, hedonic capacity or neural correlates of consciousness (if we find such a thing).

A Critical Review of Open Philanthropy’s Bet On Criminal Justice Reform

Criminals should be punished in proportion to an estimate of the harm they have caused, times a factor to account for a less than 100% chance of getting caught, to ensure that crimes are not worth it in expectation.


Where is this perspective coming from? I googled this and it seems to be referring to rational choice theory in criminology. How well supported is this theory?

This is coming from my own understanding of what is politically feasible. I think that policies that make crime worth it in expectation are likely to be politically very unpopular and/or lead to more crime. So I think that restorative justice approaches would be stronger with a punitive component, even if they overall reduce that punitive component.
The totalitarian implications of Effective Altruism

. My concern however is that this is not EA, or at least not EA as embodied by its fundamental principles as explored in my piece. 


And I think your last para is mostly valid too.

What are EA's biggest legible achievements in x-risk?

Thanks for this anecdote!

Given the scarcity of such successes, I think people here would be interested in hearing a longer form version of this. Just wished to suggest!

What are EA's biggest legible achievements in x-risk?

Thanks for your reply! I'll see if I can convince people using this.

(Also very small point but: the pdf title says "Insert your title here" when viewed on chrome atleast)

What are EA's biggest legible achievements in x-risk?

Thanks for your reply!

This makes sense. Linked post at the end was useful and new for me.

Catholic theologians and priests on artificial intelligence

Would recommend adding something attention catching or interesting from the special issue if you can. Will increase the likelihood someone decides its worth their time investment to go through it.

Good call -- I added a little more detail about these two discussions.
The totalitarian implications of Effective Altruism

Nice article.

One mental move you can make to avoid this totalisation is to frame "doing the most good" not as the terminal goal of your life, but as an instrumental goal to your terminal goal which is "be happy/satisfied/content". Doing good is obviously one of the things that brings you satisfaction (if it didn't you obviously would not do it), but it isn't the only thing.

Accepting this frame risks ending up not doing good at all, cause there's plenty of people who are happy without doing much good for others (atleast as measured by an EA lens). Which may... (read more)

Thanks for engaging with my piece and for these interesting thoughts - really appreciate it. I agree that, on a personal level, turning 'doing the most good' into an instrumental goal towards the terminal goal of 'being happy' sounds like an intuitive and healthy way to approach decision-making. My concern however is that this is not EA, or at least not EA as embodied by its fundamental principles as explored in my piece. The question that comes to my mind as I read your comment is: 'is instrumental EA (A) a personal ad hoc exemption to EA (i.e. a form of weak EA), or (B) a proposed reformulation of EA's principles?' If the former, then I think this is subject to the same pressures as outlined in my piece. If the latter, then my concern would be that the fundamental objective of this reformulation is so divorced from EA's original intention that the concept of EA becomes meaningless.
AGI Ruin: A List of Lethalities

Interesting suggestion, although my first reaction is it feels a bit like handing things over to Moloch. Like, I would rather replace a bad judge of what is infohazardous content with a good judge, than lose our ability to keep any infohazards private at all.

There's also a similar discussion on having centralized versus decentralized grantmaking for longterm future stuff. People pointed out unilateralists curse as a reason to keep it centralised.

I am normally super in favour of more decentralisation and pluralism of thought, but having seen a bunch of info... (read more)

2Charles He23d
This is another one of those downvoted comments. This comment was at -2 until I strong upvoted. :/
AGI Ruin: A List of Lethalities

Just as a datapoint:

I had once explicitly posted on LW asking what pivotal act you could take by querying a misaligned oracle AI, with the assumption that you want to leak as little bits of information about the world as possible to the AI. Reasoning being if it had lesser data it would have lesser ability to break out of its box even if it failed to answer your query.

LW promptly downvoted the question heavily so I assumed it's taboo.

This makes me wish there were a popular LW- or EAF-ish forum that doesn't use the karma system (or uses a very different karma system). If karma sometimes makes errors sticky because disagreement gets downvoted, then it would be nice to have more venues that don't have exactly that issue. (Also, if more forums exist, this makes it likelier that different forums will downvote different things, so a wider variety of ideas can be explored in at least one place.) This is also another reason to roll out separate upvotes for 'I like this' versus 'I agree with this'.
Breaking Up Elite Colleges

Ignoring inside views on specific topics, yes we should have that prior. But having inside views on both AI risk and stable totalitarianism (without use of AGI), I'm personally leaning towards net negative currently.

Safety work on either AI risk or biorisk or stable totalitarianism doesn't seem as limited by the wealth civilisation as a whole has, as it does by the number of people who agree and care enough direct funds, attention or energy to such causes.

Breaking Up Elite Colleges

a) It isn't obvious undifferentiated scientific progress is net good, or any interventions to speed it up. AI capabilities will also be sped up which seems net bad. Even in futures where AI risk is not an issue (we don't build an AGI), scientific progress being sped up might bring closer the possibility of stable totalitarianism, and black balls our society is currently ill-prepared to tackle.

b) EA disproprotionately originates and hires from elite colleges. IMO this means you might have to come up with really good arguments if you want to convince them. Just wished to keep you aware.

it's not obvious that undifferentiated scientific progress is net bad either. Scientific progress increases our wealth and allows for us to spend a larger fraction on safety than we otherwise would. I'd much prefer to live in a world where we can afford both nukes and safety measures, than the world in which we only could afford the nukes. Scientific progress has been the root of so much progress, I think we should have a strong prior that more of it is good!
My GWWC donations: Switching from long- to near-termist opportunities?

Trying to downplay rather than promote the visibility of AI risk feels to me at gut level like a losing strategy.

There are real benefits to having more people convinced about AI risk, such as more funding and talent dedicated to the problem. As you mention there are downsides if you fail to convince someone. There are ways to get higher upside to downside ratio when spreading ideas - like only focussing on newly graduated AI researchers. But if the ideas ever hit mainstream, it will be hard to control who does or not get exposed to them. What we terminally... (read more)

Global health is important for the epistemic foundations of EA, even for longtermists

Might be worth making this distinction more prominent in the post! I didn't notice it on first (brief) read either.

Unflattering reasons why I'm attracted to EA

Small point but the linked tweet in your last para doesn't come across as someone who feels EAs are morally arrogant, atleast if I read the thread without any other context. He's both appreciative and critical of EA, and his criticisms seem mostly on the actual work rather than attitudes or traits of the people involved.

Announcing a contest: EA Criticism and Red Teaming

I missed this, my bad.

It still invalidates the specific critique - but yeah if Pablo's point is just about "quality of critique" then this doesn't really invalidate that.

Announcing a contest: EA Criticism and Red Teaming

80k hours article on voting does not say don't vote. Do atleast link the article in your post!

pablo is quoting a 10-year-old comment; the 80k article your link was published in 2020.

Institution design for exponential technology

Im not sure how to upload a doc on phone so I'm just copy pasting the content.

If this isn't valuable for the forum I'm also happy to take it down (either the comment or the post)

This is mostly a failed attempt at doing anything useful, that I nevertheless wish to record.

See also:

There is a lot of complexity to understanding how tech changes the landscape of power and how bureaucracies can or should function. A lot of this complexity is specific to th... (read more)

Institution design for exponential technology

Thanks for your reply. I did in fact realise the same thing later on! I could send you my attempt or link it here if you're interested.

2Aman Patel1mo
Thanks for posting your attempt! Yeah, it does seem like you ran into some of those issues in your attempt, and it's useful information to know that this task is very hard. I guess one lesson here is that we probably won't be able to build perfect institutions on the first try, even in safety-critical cases like AGI governance.
Im not sure how to upload a doc on phone so I'm just copy pasting the content. If this isn't valuable for the forum I'm also happy to take it down (either the comment or the post) -------------------------------------------------------------------------------- This is mostly a failed attempt at doing anything useful, that I nevertheless wish to record. See also: [] There is a lot of complexity to understanding how tech changes the landscape of power and how bureaucracies can or should function. A lot of this complexity is specific to the tech itself. For instance, nuances of how nukes must be distributed can affect how its command structure can look like, or how the uranium supply chain must look like. Or nuances of what near-term AI cannot or cannot do, for surveillance and enforcement, can affect the structure of bureaucracies that wish to wield this power. Or nuances of how compute governance can work, for AGI governance structures. I wondered if it possible to abstract away all of this complexity by simply assuming any tech allows for a relation of power between two people, one exerts control on the other. And then build theory that is invariant to all possible relations of power and therefore all possible technologies we may invent in the future. Toy example I wished to build theory for how institutions can hold very powerful tech. Here’s a toy example I thought of. A magical orb with following properties Wearer can wear it for 10 seconds, and instantaneously make a wish that destroys all other orbs still being formed in the universe. Wearer can wear it for 60 seconds, and become an all-knowing all-intelligent all-capable god. Also assume: Anybody anywhere in the universe can start chanting (publicly known) prayers for 60 seconds to create a
Try to sell me on [ EA idea ] if I'm [ person with a viewpoint ]

I suggested a similar idea here, although my posts wasn't as clear:

Also I think [person with a viewpoint] needs to be narrowed down to the few most popular viewpoints in the world or atleast developed world, otherwise this becomes a very large task.

Organizational alignment

Not sure why you got downvoted. First para is valid, second seems a bit off context. (Like yes, it's related but is it related enough to actually further the goals of the OP?)

7Rob Mitchell2mo
Well, it looks like I'm hijacking a thread about organisational scaling with some anxieties around referring to people in overly utilitarian ways that I've talked about elsewhere. Which is fair enough; interestingly I've done the opposite and talked about org scaling on threads that were fairly tangentially related and got quite a few upvotes for it. All very intriguing and if you're not occasionally getting blasted, you're not learning as much as you might, getting enough information about e.g. limits, etc...
a UBI-generating currency: Global Income Coin

Understood. Sorry for harshness in my original comment then.

a UBI-generating currency: Global Income Coin

Note: OP has assured me the following is not a concern. Haven't deleted the comment so the discussion is still visible.

Potential to be deceptive

I'm using a large heading because I feel the cost of EA supporting fraudulent projects can be high; I don't usually do this.

As someone who has previously worked in the cryptocurrency space, I don't think your stablecoin is likely to maintain peg longterm unless it has significant underlying reserves - USD or other assets. Significant here means atleast 40-50% backing, although it is much better if it higher. You ma... (read more)

Oh don't get me wrong. We're not a stablecoin, don't maintain a peg, and won't make a promise of value or stability. GLO is a floating currency; its value will need to be based on usage and not on any backing. This is deliberate: in the long term we believe a floating currency is more feasible than keeping a peg to an existing currency. This means there will indeed be volatility and this is a risk for anyone adopting GLO. I realize the forum post may make it sound like we're suggesting a peg/stablecoin. That's not our intention. We didn't go into detail because we didn't want to make the post too technical, but on our homepage [] and whitepaper [] we're very explicit about this. We do operate a reserve, and have a trading strategy [] that should, over time, grow the reserves as demand for GLO rises. But the goal of the reserves is merely to dampen volatility. In these interventions, we use $1 as a loose target, not as a peg. (Feedback on this trading strategy is very welcome.)
Did Peter Thiel give "the keynote address at an EA conference"?

Peter Thiel's address to EA in 2013, San Francisco:

I see. Rather than being "the" keynote speaker in 2013, there were four keynote speakers of which Thiel was one (the others were Peter Singer, Jaan Tallinn, and Holden Karnofsky.)

Messy personal stuff that affected my cause prioritization (or: how I started to care about AI safety)

My concern about people and animals having net-negative lives has been related to what’s happening with my own depression. 

+1 on this part.

I feel like this is a very common failure mode - people look at the happiness or suffering prevalent in their own personal life and extrapolate it to whether the average person is net positive, zero or negative utility.

1Tobias Dänzer2mo
Also see Wikipedia on affective forecasting and projection bias []. (h/t the book Stumbling on Happiness [Also see Wikipedia on affective forecasting and projection bias. (h/t the book Stumbling on Happiness; and I'm not sure whether this field of study was hit by the replication crisis)]; note that I'm not sure whether this field of study was hit by the replication crisis)

For what it's worth, my experience hasn't matched this. I started becoming concerned about the prevalence of net-negative lives during a particularly happy period of my own life, and have noticed very little correlation between the strength of this concern and the quality of my life over time. There are definitely some acute periods where, if I'm especially happy or especially struggling, I have more or less of a system-1 endorsement of this view. But it's pretty hard to say how much of that is a biased extrapolation, versus just a change in the size of my empathy gap from others' suffering.

EA frontpages should compare/contrast with other movements

Thanks for replying!

You're right there's some difficulty in figuring out which opinions do we want to poll EAs for and present data. And how much would outsiders trust that data, as opposed to perceptions they may have obtained through other means. I didn't fully think this through when I was posting.

re: PR department

I think it would be useful to have PR, although I'm not entirely sure what policies they should take, or how insider versus outsider conversations should be fragmented. Some people have raised issue with the EA forum too because it is too open... (read more)

6Luke Chambers2mo
No problem! Absolutely, I can see what you mean. Personal lens counts for a lot and people can run away with ideas of badness. Things are rarely as bad as people criticise them for being, and EA is no different. Yeah it has a few issues here and there but the media and Twitter can often make these issues look far, far worse than they actually are. I can totally see where you're coming from.
EA frontpages should compare/contrast with other movements

Hmm, maybe I can call it a prior but from the view of the outsider. Say an outsider with far-left stances comes and talks to 5 EAs, and finds 1 person who shares their far-left opinions, and 1 person who has stances they find repulsive. Or maybe it's not 5 EAs in person they meet, but 5 articles or blog posts they read. They're now going to assume that roughly 20% of EAs are far-left and 20% have opinions they find repulsive, by extrapolating their "prior".

(And sure the ideal thing to do is talk to more than 5 people but not everyone has time or interest for that, which is why I might want EAs to instead present this data to them explicitly.)

EA frontpages should compare/contrast with other movements

I see - this is a valid point.

I was thinking of reporting survey results (of EA opinions on non-EA stances) - do you think it is hard to conduct surveys objectively?

In case you haven't seen this, it might address some of (though likely not all of) what you're looking for:
Surveys seem valuable, I'd certainly find the results interesting.
EA frontpages should compare/contrast with other movements

You're right "bias" may not have been the best word for what I wanted to say, it's somewhat negatively-coded.

I'd be keen to know more what you mean by "what directs one to arrive at such a position".

1Jay Bailey2mo
A better word for this might be "priors" - it indicates that there's already existing inertia towards these positions in EA, but doesn't carry the connotation of this being a bad thing. That said, it's not 100% accurate - a "prior" is an assumption of the world that comes from deep-seated beliefs or upbringing, whereas many of us in EA who have certain existing stances have done so via reasoning.
EA frontpages should compare/contrast with other movements

2 makes sense.

Regarding 3, I completely agree. I think you can present this nuance in the intro pages too. Like here is the current distribution of EA opinions on your favorite movement X, but if you feel you can convince us on X we're open to change.

EA frontpages should compare/contrast with other movements

Thank you for this! This makes sense, and this could be added to intro pages.

I also still think it's useful to list out (in intro pages) systemic changes that other movements support that we don't support (or atleast don't have consensus support for inside of EA).

Load More