I haven't had time to read all the discourse about Manifest (which I attended), but it does highlight a broader issue about EA that I think is poorly understood, which is that different EAs will necessarily have ideological convictions that are inconsistent with one another.  That is, some people will feel their effective altruist convictions motivate them to work to build artificial intelligence at OpenAI or Anthropic; others will think those companies are destroying the world. Some will try to save lives by distributing medicines; others will think the people those medicines save eat enough tortured animals to generally make the world worse off. Some will think liberal communities should exclude people who champion the existence of racial differences in intelligence; others will think excluding people for their views is profoundly harmful and illiberal.  I'd argue that the early history of effective altruism (i.e. the last 10-15 years) has generally been one of centralization around purist goals -- i.e. there're central institutions that effective altruism revolves around and specific causes and ideas that are the most correct form of effective altruism. I'm personally much more a proponent of liberal, member-first effective altruism than purist, cause-first EA. I'm not sure which of those options the Manifest example supports, but I do think it's indicative of the broader reality that for a number of issues, people on each side can believe the most effective altruist thing to do is to defeat the other. 

Let Manifest be Manifest. Having a space that is intellectually-edgy, but not edge-lord maxing seems extremely valuable. Especially given how controversial some EA ideas were early on (and how controversial wild animal welfare and AI welfare still are). In fact, I'd go further and suggest that it would be great if they were to set up their own forum. This would allow us to nudge certain discussions into an adjacent, not-explicitly EA space instead of discussing it here. Certain topics are a poor fit for the forum because they rate high on controversy but low-but-non-zero on relevance EA. At the same time, I wouldn't want to completely shut down such discussions, because sometimes things turn out to be more important than you thought when you dive into the details. So I guess I'd really love to see another non-EA space end up being the first port of call for such discussions, with the hope that only the highest quality and most relevant ideas would make it over to the EA forum.

Just a prompt to say that if you've been kicking around an idea of possible relevance to the essay competition on the automation of wisdom and philosophy, now might be the moment to consider writing it up -- entries are due in three weeks.

Manifest 2025 https://youtu.be/FVvg1CKBE20

A piece of career advice I've given a few times recently to people in AI Safety, which I thought worth repeating here, is that AI Safety is so nascent a field that the following strategy could be worth pursuing: 1. Write your own job description (whatever it is that you're good at / brings joy into your life). 2. Find organisations that you think need thing that job but don't yet have it. This role should solve a problem they either don't know they have or haven't figured out how to solve. 3. Find the key decision maker and email them. Explain the (their) problem as you see it and how having this role could fix their problem. Explain why you're the person to do it. I think this might work better for mid-career people, but, if you're just looking to skill up and don't mind volunteering, you could adapt this approach no matter what stage you're at in your career.

Leopold Aschenbrenner is starting a cross between a hedge fund and a think tank for AGI. I have read only the sections of Situational Awareness most relevant to this project, and I don't feel nearly like I understand all the implications, so I could end up being quite wrong. Indeed, I’ve already updated towards a better and more nuanced understanding of Aschenbrenner's points, in ways that have made me less concerned than I was to begin with. But I want to say publicly that the hedge fund idea makes me nervous. Before I give my reasons, I want to say that it seems likely most of the relevant impact comes not from the hedge fund but from the influence the ideas from Situational Awareness have on policymakers and various governments, as well as the influence and power Aschenbrenner and any cohort he builds wield. This influence may come from this hedge fund or be entirely incidental to it. I mostly do not address this here, but it does make all of the below less important.  I also believe that some (though not all) of my concerns about the hedge fund are based on specific disagreements with Aschenbrenner’s views. I discuss some of those below, but a full rebuttal this is not (and many of the points of disagreement I don’t yet feel confident in my view on). There is still plenty to do to hash out the actual empirical questions at hand. Why I am nervous  A hedge fund investing in AI related investments means Aschenbrenner and his investors will gain financially from more and accelerated AGI progress. This seems to me to be one of the most important dynamics (excluding the points about influence above). That creates an incentive to create more AGI progress, even at the cost of safety, which seems quite concerning. I will say that Leopold has a good track record here around turning down money in not signing an NDA at Open AI despite loss of equity. Aschenbrenner expresses strong support for the liberal democratic world to maintain a lead on AI advancement, and ensure that China does not reach an AI-based decisive military advantage over the United States[1]. The hedge fund, then, presumably aims to both support the goal of maintaining an AI lead over China and profit off of it. In my current view, this approach increases race dynamics and increases the risks of the worst outcomes (though my view on this has softened somewhat since my first draft, for reasons similar to what Zvi clarifies here[2]).  I especially think that it risks unnecessary competition when cooperation - the best outcome - could still be possible. It seems notable, for example, that no Chinese version of the Situational Awareness piece has come to my attention; going first in such a game both ensures you are first and that the game is played at all.  It’s also important that the investors (e.g. Patrick Collison) appear to be more focused on economic and technological development, and less concerned about risks from AI. The incentives of this hedge fund are therefore likely to point towards progress and away from slowing down for safety reasons.  There are other potential lines of thought here I have not yet fleshed out including:  * The value of aiming to orient the US government and military attention to AGI (seems like a huge move with unclear sign) * The degree to which this move is unilateralist on Aschenbrenner’s part * How much money could be made and how much power the relevant people (e.g. Aschenbrenner and his investors) will have through investment and being connected to important decisions.  * If a lot of money and/or power could be acquired, especially over AGI development, then there’s a healthy default skepticism I think should be applied to their actions and decision-making.  * Specifics about Aschenbrenner himself. Different people in the same role would take very different actions, so specifics about his views, ways of thinking, and profile of strengths and weaknesses may be relevant. Ways that the hedge fund could in fact be a good idea: EA and AI causes could really use funder diversification. If Aschenbrenner intends to use the money he makes to support these issues, that could be very valuable (though I’ve certainly become somewhat more concerned with moonshot “become a billionaire to save the world” plans than I used to be). The hedge fund could position Aschenbrenner to have a deep understanding of and connections within the AI landscape, making the think tank outputs very good, and causing important future decisions to be made better.  Aschenbrenner of course could be right about the value of the US government’s involvement, maintaining a US lead, and the importance of avoiding Chinese military supremacy over the US. In that case, him achieving his goals would of course be good. Cruxes include the likelihood of international cooperation, the possibility of international bans, probability of catastrophic outcomes from AI and the likelihood of “muddling through” on alignment. I’m interested in hearing takes, ways I could be wrong, fleshing out of my arguments, or any other thoughts people have relevant to this. Happy to have private chats in DMs to discuss as well. 1. ^  To be clear, Aschenbrenner wants that lead to exist to avoid a tight race in which safety and caution are thrown to the winds. If we can achieve that lead primarily through infosecurity (something he emphasizes), then added risks are low; but I think the views expressed in Situational Awareness also imply the importance of staying technologically ahead of China as their AI research improves. This comes with precisely the risks of creating and accelerating a race of this nature. Additionally, when I read his description of the importance of even a two month lead, it implied to me that if the longer, more comfortable lead is lost, there will be strong reasons for the US to advance quickly so as to avoid China reaching superintelligence and subsequent military dominance first (which doesn’t mean he thinks we should actually do this if the time came). This seems to fairly explicitly describe the tight race scenario. I don’t think Aschenbrenner believes this would be a good situation to be in, but nonetheless thinks that’s what the true picture is.  2. ^ From Zvi’s post: “He confirms he very much is NOT saying this: The race to ASI is all that matters. The race is inevitable. We might lose. We have to win. Trying to win won’t mean all of humanity loses. Therefore, we should do everything in our power to win. I strongly disagree with this first argument. But so does Leopold.  Instead, he is saying something more like this: ASI, how it is built and what we do with it, will be all that matters. ASI is inevitable. A close race to ASI between nations or labs almost certainly ends badly. Our rivals getting to ASI first would also be very bad. Along the way we by default face proliferation and WMDs, potential descent into chaos. The only way to avoid a race is (at least soft) nationalization of the ASI effort. With proper USG-level cybersecurity we can then maintain our lead.  We can then use that lead to ensure a margin of safety during the super risky and scary transition to superintelligence, and to negotiate from a position of strength.”  

I quit. I'm going to stop calling myself an EA, and I'm going to stop organizing EA Ghent, which, since I'm the only organizer, means that in practice it will stop existing. It's not just because of Manifest; that was merely the straw that broke the camel's back. In hindsight, I should have stopped after the Bostrom or FTX scandal. And it's not just because they're scandals; It's because they highlight a much broader issue within the EA community regarding whom it chooses to support with money and attention, and whom it excludes. I'm not going to go to any EA conferences, at least not for a while, and I'm not going to give any money to the EA fund. I will continue working for my AI safety, animal rights, and effective giving orgs, but will no longer be doing so under an EA label. Consider this a data point on what choices repel which kinds of people, and whether that's worth it. EDIT: This is not a solemn vow forswearing EA forever. If things change I would be more than happy to join again.

If you’re seeing things on the forum right now that boggle your mind, you’re not alone. Forum users are only a subset of the EA community. As a professional community builder, I’m fortunate enough to know many people in the EA community IRL, and I suspect most of them would think it’d be ridiculous to give a platform to someone like Hanania. If you’re like most EAs I know, please don’t be dissuaded from contributing to the forum. I’m very glad CEA handles its events differently.

AI Safety Needs To Get Serious About Chinese Political Culture I worry that Leopold Aschenbrenner's "China will use AI to install a global dystopia" take is based on crudely analogising the CCP to the USSR, or perhaps even to American cultural imperialism / expansionism, and isn't based on an even superficially informed analysis of either how China is currently actually thinking about AI, or what China's long term political goals or values are. I'm no more of an expert myself, but my impression is that China is much more interested in its own national security interests and its own ideological notions of the ethnic Chinese people and Chinese territory, so that beyond e.g. Taiwan there isn't an interest in global domination except to the extent that it prevents them being threatened by other expansionist powers. This or a number of other heuristics / judgements / perspectives could change substantially how we think about whether China would race for AGI, and/or be receptive to an argument that AGI development is dangerous and should be suppressed. China clearly has a lot to gain from harnessing AGI, but they have a lot to lose too, just like the West. Currently, this is a pretty superficial impression of mine, so I don't think it would be fair to write an article yet. I need to do my homework first: * I need to actually read Leopold's own writing about this, instead of making impressions based on summaries of it, * I've been recommended to look into what CSET and Brian Tse have written about China, * Perhaps there are other things I should hear about this, feel free to make recommendations. Alternatively, as always, I'd be really happy for someone who's already done the homework to write about this, particularly anyone specifically with expertise in Chinese political culture or international relations. Even if I write the article, all it'll really be able to be is an appeal to listen to experts in the field, or for one or more of those experts to step forward and give us some principles to spread in how to think clearly and accurately about this topic. I think having even like, undergrad-level textbook mainstream summaries of China's political mission and beliefs posted on the Forum could end up being really valuable if it puts those ideas more in the cultural and intellectual background of AI safety people in general. This seems like a really crucial question that inevitably takes a central role in our overall strategy, and Leopold's take isn't the only one I'm worried about. I think people are already pushing national security concerns about China to the US Government in an effort to push e.g. stronger cybersecurity controls or export controls on AI. I think that's a noble end but if the China angle becomes inappropriately charged we're really risking causing more harm than good. (For the avoidance of doubt, I think the Chinese government is inhumane, and that all undemocratic governments are fundamentally illegitimate. I think exporting democracy and freedom to the world is a good thing, so I'm not against cultural expansionism per se. Nevertheless, assuming China wants to do it when they don't could be a really serious mistake.)

Have your EA conflicts on... THE FORUM! In general, I think it's much better to first attempt to have a community conflict internally before I have it externally. This doesn't really apply to criminal behaviour or sexual abuse. I am centrally talking about disagreements, eg the Bostrom stuff, fallout around the FTX stuff, Nonlinear stuff, now this manifest stuff.  Why do I think this? * If I want to credibly signal I will listen and obey norms, it seems better to start with a small discourse escalation rather than a large one. Starting a community discussion on twitter is like jumping straight to a shooting war.  * Many external locations (eg twitter, the press) have very skewed norms/incentives to the forum and so many parties can feel like they are the victim. I find when multiple parties feel they are weaker and victimised that is likely to cause escalation.  * Many spaces have less affordance for editing comments, seeing who agrees with who, having a respected mutual party say "woah hold up there" * It is hard to say "I will abide by the community sentiment" if I have already started the discussion elsewhere in order to shame people. And if I don't intend to abide by the community sentiment, why am I trying to manage a community conflict in the first place. I might as well just jump straight to shaming.  * It is hard to say "I am open to changing my mind" if I have set up the conflict in a way that leads to shaming if the other person doesn't change theirs. It's like holding a gun to someone's head and saying that this is just a friendly discussion.  * I desire reconciliation. I have hurt people in this community and been hurt by them. In both case to the point of tears and sleepless night. But still I would prefer reconciliation and growth over a escalating conflict * Conflict is often negative sum, so lets try and have it be the least negative sum as possible.  * Probably a good chunk of it is church norms, centred around 1 Corinthians 6[2]. I don't really endorse this, but I think it's good to be clear why I think thinks.  Personal examples: * Last year I didn't like that Hanania was a main speaker at manifest (iirc) so I went to their discord and said so. I then made some votes. The median user agreed with me and so Hanania didn't speak. I doubt you heard about this, because I did it on the manifold discord. I hardly tweeted about it or anything. This and the fact I said I wouldn't created a safe space to have the discussion and I largely got what I wanted.  You might think this is a comment is directed at a specific person, but I bet you are wrong. I dislike this behaviour when it is done by at least 3 different parties that I can think of.  1. ^   2. ^  If any of you has a dispute with another, do you dare to take it before the ungodly for judgment instead of before the Lord’s people? 2 Or do you not know that the Lord’s people will judge the world? And if you are to judge the world, are you not competent to judge trivial cases? 3 Do you not know that we will judge angels? How much more the things of this life! 4 Therefore, if you have disputes about such matters, do you ask for a ruling from those whose way of life is scorned in the church? 5 I say this to shame you. Is it possible that there is nobody among you wise enough to judge a dispute between believers? 6 But instead, one brother takes another to court—and this in front of unbelievers! 7 The very fact that you have lawsuits among you means you have been completely defeated already. Why not rather be wronged? Why not rather be cheated?

When AI Safety people are also vegetarians, vegans or reducetarian, I am pleasantly surprised, as this is one (of many possible) signals to me they're "in it" to prevent harm, rather than because it is interesting.

Sharing a piece of advice I've given to a few people about applying for (EA) funding. I've heard various people working on early-stage projects express hesitancy about applying for EA funding because their plan isn't "complete" enough. They don't feel confident enough in their proposal, or think what they're asking for is too small. They seem to assume that EA funders only want to look at proposals with a long time-horizons from applicants who will work full-time who are confident their plan will work. In my experience (I've done various bits of grantmaking and regularly talk to EA funders), grantmakers in EA spaces are generally happy to receive applications that don't have these qualities. It's okay to apply if you just want to test a project out for a few months; maybe you won't be full-time, maybe you aren't confident in some part of the theory of change, maybe it's just a few months. You should apply and just explain your thinking, including all of your uncertainties. Funders are uncertain too, and often prefer to fund tests for a few months than commit to multi-year projects with full-time staff because tests give them useful information about you and the theory of change. Ideally, funders eventually support long-term projects too. I'm not super confident in this take, but I ran it past a few EA funders and they agreed. Note that I think this probably doesn't apply outside of EA; I understand many grant applications require detailed plans.

For those voting in the EU election and general elections in Belgium, here's an overview of the party positions when it comes to animal welfare: (For more details, click this link) ✅ means more in favor    ❌ means more against Federal election (Flanders): policy proposal PVDA 🔴 GROEN ❇️ VOORUIT 🔺 Open-VLD 🔵 CD&V 🔶 N-VA 🔆 VB ⬛️ VAT rate reduction on veterinary care and pet food✅✅❌❌❌❌✅A ban on traditional fireworks✅✅✅❌❌✅✅ Federal election (Walloon): policy proposal PTB 🔴 ECOLO ❇️ PS 🔺 LE 🐬 Défi 🌸 MR 🔵 VAT rate reduction on veterinary care and pet food✅❌✅❌❌✅A ban on traditional fireworks✅❌✅❌❌✅ Flanders election: policy proposal PVDA 🔴 GROEN ❇️ VOORUIT 🔺 Open-VLD 🔵 CD&V 🔶 N-VA 🔆 VB ⬛️ Better living conditions for broiler chickens in Flanders✅✅✅❌❌✅✅A ban on live cooking and cutting lobsters in half✅✅❌❌❌❌❌A phasing out plan of Boudewijn Seapark✅✅✅❌❌✅✅A ban on the painful surgical castration of piglets✅✅✅❌❌✅❌A ban on chick killing✅✅✅✅❌✅✅Stricter legislation around the dog and cat trade✅✅✅❌❌✅✅A duty of care for horses, dogs, cats and rabbits✅✅❌❌❌✅✅The development of cultured meat in Flanders✅✅✅✅❌✅❌Animal testing: for an animal-free strategy in Flanders✅✅✅✅❌✅✅A Flemish ban on the sale of products that harm animal welfare✅✅✅❌❌✅❌Animal welfare as a criterion in environmental permit procedure✅✅❌❌❌✅❌A punishment of animal abuse through GAS fines ❌✅❌❌❌✅✅total score11/1212/128/123/120/1211/127/12Highest score ✅      Walloon election:   PTB 🔴 ECOLO ❇️ PS 🔺 LE 🐬 Défi 🌸 MR 🔵 Total score12/1311/138/1310/136/135/13Highest score✅      EU election (Flanders):   PVDA 🔴 GROEN ❇️ VOORUIT 🔺 Open-VLD 🔵 CD&V 🔶 N-VA 🔆 VB ⬛️ Total score9/1010/1010/108/100/1010/100/10Highest score ✅✅  ✅  EU election (Walloon):   PTB 🔴 ECOLO ❇️ PS 🔺 LE 🐬 Défi 🌸 MR 🔵 Total score9/109/107/109/106/107/10Highest score✅✅ ✅   Brussels election:   PVDA 🔴 ECOLO ❇️ GROEN ❇️ PS 🔺 VOORUIT 🔺 LE 🐬 Défi 🌸 MR 🔵 O-VLD 🔵 CD&V 🔶 N-VA 🔆 VB ⬛️ Score5/65/66/64/66/65/65/64/63/60/66/65/6Highest score  ✅ ✅     ✅  Highest score Federal election (Flanders):  PVDA, GROEN, VB Highest score Federal election (Walloon):   PTB, PS, MS Highest score Flanders election:                   GROEN Highest score Walloon election:                    PTB Highest score EU election (Flanders):         GROEN, Vooruit, N-VA Highest score EU election (Walloon):          PTB, Ecolo, LE Highest score Brussels election:                  GROEN, Vooruit, N-VA TLDR: Just like my post on the topic pointed out, the leftwing parties tend to be best for animal welfare, but the far-right can often be better than the center-right

I was thinking about to what extent NDAs (either non-disclosure or non-disparagement agreements) played a role in the 2018 blowup at Alameda Research (since if there were a lot, that could be a throughline between messiness at Alameda and messiness at Open AI recently). Here's what I've collected from public records: * Not mentioned as far as I can tell in Going Infinite * Ben West: "I don’t want to speak for this person, but my own experience was pretty different. For example: Sam was fine with me telling prospective AR employees why I thought they shouldn’t join (and in fact I did do this),[4] and my severance agreement didn’t have any sort of non-disparagement clause. This comment says that none of the people who left had a non-disparagement clause, which seems like an obvious thing a person would do if they wanted to use force to prevent disparagement.[5]" From here * Kerry Vaughn: "Information about pre-2018 Alameda is difficult to obtain because the majority of those directly involved signed NDAs before their departure in exchange for severance payments. I am aware of only one employee who did not. The other people who can spreak freely on the topic are early investors in Alameda and members of the EA community who heard about Alameda from those directly involved before they signed their NDAs". From here. * ftxthrowaway: "Lastly, my severance agreement didn't have a non-disparagement clause, and I'm pretty sure no one's did. I assume that you are not hearing from staff because they are worried about the looming shitstorm over FTX now, not some agreement from four years ago." From here (it's a response to the previous) * nbouscal: “I'm the person that Kerry was quoting here, and am at least one of the reasons he believed the others had signed agreements with non-disparagement clauses. I didn't sign a severance agreement for a few reasons: I wanted to retain the ability to sue, I believed there was a non-disparagement clause, and I didn't want to sign away rights to the ownership stake that I had been verbally told I would receive. Given that I didn't actually sign it, I could believe that the non-disparagement clauses were removed and I didn't know about it, and people have just been quiet for other reasons (of which there are certainly plenty).” From here (it's a response to the previous) * Later says "I do think I was probably just remembering incorrectly about this to be honest, I looked back through things from then and it looks like there was a lot of back-and-forth about the inclusion of an NDA (among other clauses), so it seems very plausible that it was just removed entirely during that negotiation (aside from the one in the IP agreement)." Link here. * arthrowaway: "Also no non-disparagement clause in my agreement. FWIW I was one of the people who negotiated the severance stuff after the 2018 blowup, and I feel fairly confident that that holds for everyone. (But my memory is crappy, so that's mostly because I trust the FB post about what was negotiated more than you do.)" From here (it's in the same thread as the above)   Overall this tells a story where NDAs weren't a big part of the Alameda story (since I think Ben West and nbouscal at least left during the 2018 blowup, but folks should correct me if I'm wrong). This is a bit interesting to me.  Interested in if others have different takeaways.

For me, perhaps the biggest takeaway from Aschenbrenner's manifesto is that even if we solve alignment, we still have an incredibly thorny coordination problem between the US and China, in which each is massively incentivized to race ahead and develop military power using superintelligence, putting them both and the rest of the world at immense risk. And I wonder if, after seeing this in advance, we can sit down and solve this coordination problem in ways that lead to a better outcome with a higher chance than the "race ahead" strategy and don't risk encountering a short period of incredibly volatile geopolitical instability in which both nations develop and possibly use never-seen-before weapons of mass destruction. Edit: although I can see how attempts at intervening in any way and raising the salience of the issue risk making the situation worse.

EA organizations frequently ask for people to run criticism by them ahead of time. I’ve been wary of the push for this norm. My big concerns were that orgs wouldn’t comment until a post was nearly done, and that it would take a lot of time. My recent post  mentioned a lot of people and organizations, so it seemed like useful data. I reached out to 12 email addresses, plus one person in FB DMs and one open call for information on a particular topic.  This doesn’t quite match what you see in the post because some people/orgs were used more than once, and other mentions were cut. The post was in a fairly crude state when I sent it out. Of those 14: 10 had replied by the start of next day. More than half of those replied within a few hours. I expect this was faster than usual because no one had more than a few paragraphs relevant to them or their org, but is still impressive. It’s hard to say how sending an early draft changed things. One person got some extra anxiety because their paragraph was full of TODOs (because it was positive and I hadn’t worked as hard fleshing out the positive mentions ahead of time). I could maybe have saved myself one stressful interaction if I’d realized I was going to cut an example ahead of time Only 80,000 Hours, Anima International, and GiveDirectly failed to respond before publication (7 days after I emailed them). Of those, only 80k's mention was negative. I didn’t keep as close track of changes, but at a minimum replies led to 2 examples being removed entirely, 2 clarifications and some additional information that made the post better. So overall I'm very glad I solicited comments, and found the process easier than expected. 

Reflections 🤔 on EA & EAG following EAG London (2024): * I really liked the location this year. The venue itself was easy to get to on public transport, seemed sleek and clean, and having lots of natural light on the various floors made for a nice environment. We even got some decent sun (for London) on Saturday and Sunday. Thanks to all the organisers and volunteers involved, I know it’s lot of work setting up an event like this us and making it run smoothly. * It was good to meet people in person who I previous had only met or recognised from online interaction. I won’t single out individual 1-on-1s I had, but it was great to be able to put faces to names, and hearing peoples stories and visions in person was hugely inspiring. I talked to people involved in all sorts of cause areas and projects, and that combination of diversity, compassion, and moral seriousness is one of the best things about EA. * Listening to the two speakers from the Hibakusha Project at the closing talk was very moving, and clear case of how knowing something intellectually is not the same thing as hearing personal (and in-person) testimony. I think it would’ve been one of my conference highlights in the feedback form if we hadn’t already been asked to fill it out a few minutes beforehand! * I was going to make a point about a ‘lack of EA leadership’ turning up apart from Zach Robinson, but when I double-checked the event attendee list I think I was just wrong on this. Sure, a couple of big names didn’t turn up, and it may depend on what list of ‘EA leaders’ you’re using as a reference, but I want to admit I was directionally wrong here. * I thought Zach gave a good opening speech, but many people noted on the apparent dissonance between saying that CEA wanted to focus on ‘principles-first’ approach to EA, but that they also expected AI to be their area of most focus/highest priority and that they don't expect that to change in the near future. * Finally, while I’m sure the people I spoke to (and those who wanted to speak to me) is strongly affected by selection-effects, and my own opinions on this are fairly strong, it did feel that there was consensus on there being a lack of trust/deference/shared beliefs from ‘Bay-Area EA’:[1] * Many people think that working on AI Safety and Governance is important and valuable, but not 'overwhelmingly important' or 'the most important thing human has done/will ever do'. This included some fairly well-known names from those who attended, and basically nobody there (as far as I could tell) I interacted with held extremely 'doomer' beliefs about AI. * There was a lot of uncomfortable feeling at the community-building funding being directed to ‘longtermism’ and AI Safety in particular. This is definitely a topic I'm want to investigate more post-EAG, as I'm not sure what the truth of the matter is, but I'd certainly find it problematic if some of the anecdotes I heard were a fair representation of reality. * In any case, I think it's clear that AI Safety is no longer 'neglected' within EA, and possibly outside of it.[2] (Retracted this as, while it's not true, commenters have pointed out that it's not really the relevant metric to be tracking here)  * On a personal level, it felt a bit odd to me that the LessOnline conference was held at exactly the same time as EAG. Feels like it could be a coincidence, but on the other hand this is not a coincidence because nothing is ever a coincidence. It feeds into my impression that the Bay is not very interested in what the rest of EA has to say. * One point which I didn't get any clear answers to was 'what are the feedback mechanisms in the community to push back on this', and do such feedback mechanisms even exist? * In summary: It feels like, from my perspective, that the Bay Area/Exclusively Longtermist/AI Safety Maximalist version of EA has 'lost of the mandate of heaven', but nonetheless at the moment controls a lot of the community's money and power. This, again, is a theme I want to explicitly explore in future posts. * I am old (over 30yo) and can’t party like the young EAs anymore 😔 1. ^ I'm not sure I have a good name for this, or concrete dividing lines. But in discussions people seemed to understand what it was meant to capture. 2. ^ To me it does seem like the case for the overwhelming importance of AI has been under-argued for and under-scrutinised.

FAW#3 An interesting idea (no evidence that it would work) - just putting it here for preservation more than anything else: Insects are haraam to eat. This is obviously good news in that it means at least 20% of the global population is unlikely to contribute to the demand for insects as food. However it doesn't automatically rule-out that muslims will contribute to the demand for insects through the consumption of farmed-animals who we might use insects to feed - e.g. Chickens and Fish. It might be worth finding out if muslims would care if their chicken or fish was unnecessarily fead exclusively haraam food instead of plant-based feed. My experience as a muslim makes me feel like a lot of people would much rather prefer the animals they consume to not be fead on things which they themselves wouldn't consider halaal.

Having a baby and becoming a parent has had an incredible impact on me. Now more than ever, I feel more connected and concerned about the wellbeing of others. I feel as though my heart has literally grown. I wanted to share this as I expect there are many others who are questioning whether to have children -- perhaps due to concerns about it limiting their positive impact, among many others. But I'm just here to say it's been beautiful, and amazing, and I look forward to the day I get to talk with my son about giving back in a meaningful way.  

Pretty wild discussion in this podcast about how aggressively the USSR cut corners on safety in their space program in order to stay ahead of the US. In the author's telling of the history, this was in large part because Khrushchev wanted to rack up as many "firsts" (e.g., first satellite, first woman in space) as possible. This seems like it was most proximately for prestige and propaganda rather than any immediate strategic or technological benefit (though of course the space program did eventually produce such bigger benefits). Evidence of the following claim for AI: people may not need a reason to cut corners on safety because the material benefits are so high. They may do so just because of the prestige and glory of being first. https://www.lawfaremedia.org/article/chatter--the-harrowing-history-of-the-soviet-space-program-with-john-strausbaugh

I am not under any non-disparagement obligations to OpenAI. It is important to me that people know this, so that they can trust any future policy analysis or opinions I offer. I have no further comments at this time.

In this "quick take", I want to summarize some my idiosyncratic views on AI risk.  My goal here is to list just a few ideas that cause me to approach the subject differently from how I perceive most other EAs view the topic. These ideas largely push me in the direction of making me more optimistic about AI, and less likely to support heavy regulations on AI. (Note that I won't spend a lot of time justifying each of these views here. I'm mostly stating these points without lengthy justifications, in case anyone is curious. These ideas can perhaps inform why I spend significant amounts of my time pushing back against AI risk arguments. Not all of these ideas are rare, and some of them may indeed be popular among EAs.) 1. Skepticism of the treacherous turn: The treacherous turn is the idea that (1) at some point there will be a very smart unaligned AI, (2) when weak, this AI will pretend to be nice, but (3) when sufficiently strong, this AI will turn on humanity by taking over the world by surprise, and then (4) optimize the universe without constraint, which would be very bad for humans. By comparison, I find it more likely that no individual AI will ever be strong enough to take over the world, in the sense of overthrowing the world's existing institutions and governments by surprise. Instead, I broadly expect unaligned AIs will integrate into society and try to accomplish their goals by advocating for their legal rights, rather than trying to overthrow our institutions by force. Upon attaining legal personhood, unaligned AIs can utilize their legal rights to achieve their objectives, for example by getting a job and trading their labor for property, within the already-existing institutions. Because the world is not zero sum, and there are economic benefits to scale and specialization, this argument implies that unaligned AIs may well have a net-positive effect on humans, as they could trade with us, producing value in exchange for our own property and services. Note that my claim here is not that AIs will never become smarter than humans. One way of seeing how these two claims are distinguished is to compare my scenario to the case of genetically engineered humans. By assumption, if we genetically engineered humans, they would presumably eventually surpass ordinary humans in intelligence (along with social persuasion ability, and ability to deceive etc.). However, by itself, the fact that genetically engineered humans will become smarter than non-engineered humans does not imply that genetically engineered humans would try to overthrow the government. Instead, as in the case of AIs, I expect genetically engineered humans would largely try to work within existing institutions, rather than violently overthrow them. 2. AI alignment will probably be somewhat easy: The most direct and strongest current empirical evidence we have about the difficulty of AI alignment, in my view, comes from existing frontier LLMs, such as GPT-4. Having spent dozens of hours testing GPT-4's abilities and moral reasoning, I think the system is already substantially more law-abiding, thoughtful and ethical than a large fraction of humans. Most importantly, this ethical reasoning extends (in my experience) to highly unusual thought experiments that almost certainly did not appear in its training data, demonstrating a fair degree of ethical generalization, beyond mere memorization. It is conceivable that GPT-4's apparently ethical nature is fake. Perhaps GPT-4 is lying about its motives to me and in fact desires something completely different than what it professes to care about. Maybe GPT-4 merely "understands" or "predicts" human morality without actually "caring" about human morality. But while these scenarios are logically possible, they seem less plausible to me than the simple alternative explanation that alignment—like many other properties of ML models—generalizes well, in the natural way that you might similarly expect from a human. Of course, the fact that GPT-4 is easily alignable does not immediately imply that smarter-than-human AIs will be easy to align. However, I think this current evidence is still significant, and aligns well with prior theoretical arguments that alignment would be easy. In particular, I am persuaded by the argument that, because evaluation is usually easier than generation, it should be feasible to accurately evaluate whether a slightly-smarter-than-human AI is taking bad actions, allowing us to shape its rewards during training accordingly. After we've aligned a model that's merely slightly smarter than humans, we can use it to help us align even smarter AIs, and so on, plausibly implying that alignment will scale to indefinitely higher levels of intelligence, without necessarily breaking down at any physically realistic point. 3. The default social response to AI will likely be strong: One reason to support heavy regulations on AI right now is if you think the natural "default" social response to AI will lean too heavily on the side of laissez faire than optimal, i.e., by default, we will have too little regulation rather than too much. In this case, you could believe that, by advocating for regulations now, you're making it more likely that we regulate AI a bit more than we otherwise would have, pushing us closer to the optimal level of regulation. I'm quite skeptical of this argument because I think that the default response to AI (in the absence of intervention from the EA community) will already be quite strong. My view here is informed by the base rate of technologies being overregulated, which I think is quite high. In fact, it is difficult for me to name even a single technology that I think is currently clearly underregulated by society. By pushing for more regulation on AI, I think it's likely that we will overshoot and over-constrain AI relative to the optimal level. In other words, my personal bias is towards thinking that society will regulate technologies too heavily, rather than too loosely. And I don't see a strong reason to think that AI will be any different from this general historical pattern. This makes me hesitant to push for more regulation on AI, since on my view, the marginal impact of my advocacy would likely be to push us even further in the direction of "too much regulation", overshooting the optimal level by even more than what I'd expect in the absence of my advocacy. 4. I view unaligned AIs as having comparable moral value to humans: This idea was explored in one of my most recent posts. The basic idea is that, under various physicalist views of consciousness, you should expect AIs to be conscious, even if they do not share human preferences. Moreover, it seems likely that AIs — even ones that don't share human preferences — will be pretrained on human data, and therefore largely share our social and moral concepts. Since unaligned AIs will likely be both conscious and share human social and moral concepts, I don't see much reason to think of them as less "deserving" of life and liberty, from a cosmopolitan moral perspective. They will likely think similarly to the way we do across a variety of relevant axes, even if their neural structures are quite different from our own. As a consequence, I am pretty happy to incorporate unaligned AIs into the legal system and grant them some control of the future, just as I'd be happy to grant some control of the future to human children, even if they don't share my exact values. Put another way, I view (what I perceive as) the EA attempt to privilege "human values" over "AI values" as being largely arbitrary and baseless, from an impartial moral perspective. There are many humans whose values I vehemently disagree with, but I nonetheless respect their autonomy, and do not wish to deny these humans their legal rights. Likewise, even if I strongly disagreed with the values of an advanced AI, I would still see value in their preferences being satisfied for their own sake, and I would try to respect the AI's autonomy and legal rights. I don't have a lot of faith in the inherent kindness of human nature relative to a "default unaligned" AI alternative. 5. I'm not fully committed to longtermism: I think AI has an enormous potential to benefit the lives of people who currently exist. I predict that AIs can eventually substitute for human researchers, and thereby accelerate technological progress, including in medicine. In combination with my other beliefs (such as my belief that AI alignment will probably be somewhat easy), this view leads me to think that AI development will likely be net-positive for people who exist at the time of alignment. In other words, if we allow AI development, it is likely that we can use AI to reduce human mortality, and dramatically raise human well-being for the people who already exist. I think these benefits are large and important, and commensurate with the downside potential of existential risks. While a fully committed strong longtermist might scoff at the idea that curing aging might be important — as it would largely only have short-term effects, rather than long-term effects that reverberate for billions of years — by contrast, I think it's really important to try to improve the lives of people who currently exist. Many people view this perspective as a form of moral partiality that we should discard for being arbitrary. However, I think morality is itself arbitrary: it can be anything we want it to be. And I choose to value currently existing humans, to a substantial (though not overwhelming) degree. This doesn't mean I'm a fully committed near-termist. I sympathize with many of the intuitions behind longtermism. For example, if curing aging required raising the probability of human extinction by 40 percentage points, or something like that, I don't think I'd do it. But in more realistic scenarios that we are likely to actually encounter, I think it's plausibly a lot better to accelerate AI, rather than delay AI, on current margins. This view simply makes sense to me given the enormously positive effects I expect AI will likely have on the people I currently know and love, if we allow development to continue.

Animal Justice Appreciation Note Animal Justice et al. v A.G of Ontario 2024 was recently decided and struck down large portions of Ontario's ag-gag law. A blog post is here. The suit was partially funded by ACE, which presumably means that many of the people reading this deserve partial credit for donating to support it. Thanks to Animal Justice (Andrea Gonsalves, Fredrick Schumann, Kaitlyn Mitchell, Scott Tinney), co-applicants Jessica Scott-Reid and Louise Jorgensen, and everyone who supported this work!

Why are April Fools jokes still on the front page? On April 1st, you expect to see April Fools' posts and know you have to be extra cautious when reading strange things online. However, April 1st was 13 days ago and there are still two posts that are April Fools posts on the front page. I think it should be clarified that they are April Fools jokes so people can differentiate EA weird stuff from EA weird stuff that's a joke more easily. Sure, if you check the details you'll see that things don't add up, but we all know most people just read the title or first few paragraphs.

Please people, do not treat Richard Hannania as some sort of worthy figure who is a friend of EA. He was a Nazi, and whilst he claims he moderated his views, he is still very racist as far as I can tell. Hannania called for trying to get rid of all non-white immigrants in the US, and the sterilization of everyone with an IQ under 90 indulged in antisemitic attacks on the allegedly Jewish elite, and even post his reform was writing about the need for the state to harass and imprison Black people specifically ('a revolution in our culture or form of government. We need more policing, incarceration, and surveillance of black people' https://en.wikipedia.org/wiki/Richard_Hanania).  Yet in the face of this, and after he made an incredibly grudging apology about his most extreme stuff (after journalists dug it up), he's been invited to Manifiold's events and put on Richard Yetter Chappel's blogroll.  DO NOT DO THIS. If you want people to distinguish benign transhumanism (which I agree is a real thing*) from the racist history of eugenics, do not fail to shun actual racists and Nazis. Likewise, if you want to promote "decoupling" factual beliefs from policy recommendations, which can be useful, do not duck and dive around the fact that virtually every major promoter of scientific racism ever, including allegedly mainstream figures like Jensen, worked with or published with actual literal Nazis (https://www.splcenter.org/fighting-hate/extremist-files/individual/arthur-jensen).  I love most of the people I have met through EA, and I know that-despite what some people say on twitter- we are not actually a secret crypto-fascist movement (nor is longtermism specifically, which whether you like it or not, is mostly about what its EA proponents say it is about.) But there is in my view a disturbing degree of tolerance for this stuff in the community, mostly centered around the Bay specifically. And to be clear I am complaining about tolerance for people with far-right and fascist ("reactionary" or whatever) political views, not people with any particular personal opinion on the genetics of intelligence. A desire for authoritarian government enforcing the "natural" racial hierarchy does not become okay, just because you met the person with the desire at a house party and they seemed kind of normal and chill or super-smart and nerdy.  I usually take a way more measured tone on the forum than this, but here I think real information is given by getting shouty.  *Anyone who thinks it is automatically far-right to think about any kind of genetic enhancement at all should go read some Culture novels, and note the implied politics (or indeed, look up the author's actual die-hard libertarian socialist views.) I am not claiming that far-left politics is innocent, just that it is not racist.