Here's a question that comes to mind: if local EA communities make people 3x more motivated to pursue high-impact careers, or make it much easier for newcomers to engage with EA ideas, then even if these local groups are only operating at 75% efficiency compared to some theoretical global optimum, you still get significant net benefit.

I am sympathetic to this argument vibes wise and I thought this was an elegant numerate utilitarian case for it. Part of my motivation is that I think it would be good if a lot of EA-ish values were a lot more mainstream. Like, I would even say that you probably get non-linear returns to scale in some important ways. You kind of need a critical mass of people to do certain things. 

It feels like, necessarily, these organizations would also be about providing value to the members as well. That is a good thing.

I think there is something like a "but what if we get watered down too much" concern latent here. I can kind of see how this would happen, but I am also not that worried about it. The tent is already pretty big in some ways. Stuff like numerate utilitarianism, empiricism, broad moral circles, thoughtfulness, tough trade-offs doesn't seem in danger of going away soon. Probably EA growing would spread these ideas rather than shrink them.

Also, I just think that societies/people all over the world could significantly benefit from stronger third pillars and that the ideal versions of these sorts of community spaces would tend to share a lot of things in common with EA. 

Picture it. The year is 2035 (9 years after the RSI near-miss event triggered the first Great Revolt). You ride your bitchin' electric scooter to the EA-adjacent community center where you and your friends co-work on a local voter awareness campaign, startup idea, or just a fun painting or whatever. An intentional community. 

That sounds like a step towards the glorious transhumanist future to me, but maybe the margins on that are bad in practice and the community centers of my day dreams will remain merely EA-adjacent. Perhaps, I just need to move to a town with cooler libraries. I am really not sure what the Dao here is or where the official EA brand really fits into any of this. 

Ya, maybe. This concern/way of thinking just seems kind of niche. Probably only a very small demographic who overlaps with me here. So I guess I wouldn't expect it to be a consequential amount of money to eg. Anthropic or OpenAI.

That check box would be really cool though. It might ease friction / dissonance for people who buy into high p(doom) or relatively non-accelerationist perspectives. My views are not representative of anyone, but me, but a checkbox like that would be a killer feature for me and certainly win my $20/mo :) . And maybe, y'know, all 100 people or whatever who would care and see it that way.

Ya, really sad to hear that!

I mean, if they were going to fire him for that, maybe just don't hire him. Feels kind of mercurial that they were shamed into caning him so easily/soon.

Ofc, I can understand nonviolence being, like, an important institutional messaging northstar lol. But the vibes are kind of off when you are going to fire a recently hired podcaster bc of a cringe compilation of their show. Seriously, what the hell?

I do not think listening to that podcast made me more violent. In fact, the thought experiments and ideation like what he was touching on is, like, perfectly reasonable given the stakes he is laying out and the urgency he is advocating. Like, it's not crazy ground to cover; he talks for hours. Whatever lol, at least I think it was understandable in context.

Part of it feels like the equivalent of having to lobby against "civilian nuclear warheads". And you say "I wish that only a small nuclear detonation would occur when an accidental detonation does happen, but the fact is there's likely to be a massive nuclear chain reaction." and then getting absolutely BLASTED after someone clips you saying that you actually WISH a "small nuclear detonation" would occur. What a sadist you must be!

This feels like such stupid politics. I really want John Sherman to land on his feet. I was quite excited to see him take the role. Maybe that org was just not the right fit though...

I think it might be cool if an AI Safety research organization ran a copy of an open model or something and I could pay them a subscription to use it. That way I know my LLM subscription money is going to good AI stuff and not towards the stuff that AI companies that I don't think I like or want more of on net.

Idk, existing independent orgs might not be the best place to do this bc it might "damn them" or "corrupt them" over time. Like, this could lead them to "selling out" in a variety of ways you might conceive of that.

Still, I guess I am saying that to the extent anyone is going to actually "make money" off of my LLM usage subscriptions, it would be awesome if it were just a cool independent AIS lab I personally liked or similar. (I don't really know the margins and unit economics which seems like an important part of this pitch lol).

Like, if "GoodGuy AIS Lab" sets up a little website and inference server (running Qwen or Llama or whatever) then I could pay them the $15-25 a month I may have otherwise paid to an AI company. The selling point would be that less "moral hazard" is better vibes, but probably only some people would care about this at all and it would be a small thing. But also, it's hardly like a felt sense of moral hazard around AI is a terribly niche issue.

This isn't the "final form" of this I have in mind necessarily; I enjoy picking at ideas in the space of "what would a good guy AGI project do" or "how can you do neglected AIS / 'AI go well' research in a for-profit way".

I also like the idea of an explicitly fast follower project for AI capabilities. Like, accelerate safety/security relevant stuff and stay comfortably middle of the pack on everything else. I think improving GUIs is probably fair game too, but not once it starts to shade into scaffolding I think? I wouldn't know all of the right lines to draw here, but I really like this vibe.

This might not work well if you expect gaps to widen as RSI becomes a more important input. I would argue that seems too galaxy brained given that, as of writing, we do live in a world with a lot of mediocre AI companies that I believe can all provide products of ~comparable quality.

It is also just kind of a bet that in practice it is probably going to remain a lot less expensive to stay a little behind the frontier than to be at the frontier. And that, in practice, it may continue to not matter in a lot of cases.

Thanks for linking "Line Goes Up? Inherent Limitations of Benchmarks for Evaluating Large Language Models". Also, I agree with:

MacAskill and Moorhouse argue that increases in training compute, inference compute and algorithmic efficiency have been increasing at a rate of 25 times per year, compared to the number of human researchers which increases 0.04 times per year, hence the 500x faster rate of growth. This is an inapt comparison, because in the calculation the capabilities of ‘AI researchers’ are based on their access to compute and other performance improvements, while no such adjustment is made for human researchers, who also have access to more compute and other productivity enhancements each year.

That comparison seems simplistic and inapt for at least a few reasons. That does seem like pretty "trust me bro" justification for the intelligence explosion lol. Granted, I only listened to the accompanying podcast, so I can't speak too much to the paper.

Still, I am of two minds. I still buy into a lot of the premise of "Preparing for the Intelligence Explosion". I find the idea of getting collectively blind-sighted by rapid, uneven AI progress ~eminently plausible. There didn't even need to be that much of a fig leaf.

Don't get me wrong, I am not personally very confident in "expert level AI researcher for arbitrary domains" w/i the next few decades. Even so, it does seem like the sort of thing worth thinking about and preparing about. 

From one perspective, AI coding tools are just recursive self improvement gradually coming online. I think I understand some of the urgency, but I appreciate the skepticism a lot too.

Preparing for an intelligence explosion is a worthwhile thought experiment at least. It seems probably good to know what we would do in a world with "a lot of powerful AI" given that we are in a world where all sorts of people are trying to research/make/sell ~"a lot of powerful AI". Like just in case, at least. 

I think I see multiple sides. Lots to think about.

I think the focus is generally placed on the cognitive capacities of AIs because it is expected that it will just be a bigger deal overall. 

There is at least one 80,000 hours podcast episode on robotics. It tries to explain why it's hard to do ML on, but I didn't understand it.

Also, I think Max Tegmark wrote some stuff on slaughterbots in Life 3.0. Yikes!

You could try looking for other differential development stuff too if you want. I recently liked: AI Tools for Existential Security. I think it's a good conceptual framework for emerging tech / applied ethics stuff I think. Of course, still leaves you with a lot of questions :)
 

I love to see stuff like this!

It has been a pleasure reading this, listening to your podcast episode, and trying to really think it through,

This reminds me of a few other things I have seen lately like Superalignment, Joe Carlsmith's recent "AI for AI Safety", and the recent 80,000 Hours Podcast with Will McAskill. 

I really appreciate the "Tools for Existential Security" framing. Your example applications were on point and many of them brought up things I hadn't even considered. I enjoy the idea of rapidly solving lots of coordination failures. 

This sort of DAID approach feels like an interesting continuation on other ideas about differential acceleration and the vulnerable world hypothesis. Trying to get this right can feel like some combination of applied ethics and technology forecasting. 

Probably one of the weirdest or most exciting applications you suggest is AI for philosophy. You put it under the "Epistemics" category. I usually think of epistemics as a sub-branch of philosophy, but I think I get what you mean. AI for this sort of thing remains exciting, but very abstract to me. 

What a heady thing to think about; really exciting stuff! There is something very cosmic about the idea of using AI research and cognition for ethics, philosophy, and automated wisdom. (I have been meaning to read "Winners of the Essay competition on the Automation of Wisdom and Philosophy"). I strongly agree that since AI comes with many new philosophically difficult and ethically complex questions, it would be amazing if we could use AI to face these.

The section on how to accelerate helpful AI tools was nice too. 

Appendix 4 was gold. The DPD framing is really complimentary to the rest of the essay. I can totally appreciate the distinction you are making, but I also see DPD as bleeding into AI for Existential Safety a lot as well. Such mixed feelings. Like, for one thing, you certainly wouldn't want to be deploying whack AI in your "save the world" cutting edge AI startup. 

And it seems like there is a good case for thinking about doing better pre-training and finding better paradigms if you are going to be thinking about safer AI development and deployment a lot anyways. Maybe I am missing something about the sheer economics of not wanting to actually do pre-training ever. 

In any case, I thought your suggestions around aiming for interpretable, robust, safe paradigms were solid. Paradigm-shaping and application-shaping are both interesting.

***

I really appreciate that this proposal is talking about building stuff! And that it can be done ~unilaterally. I think that's just an important vibe and an important type of project to have going.

I also appreciate that you said in the podcast that this was only one possible framing / clustering. Although you also say "we guess that the highest priority applications will fall into the categories listed above" which seems like a potentially strong claim. 

I have also spent some time thinking about which forms of ~research / cognitive labor would be broadly good to accelerate for similar existential security reasons and I kind of tried to retrospectively categorize some notes I had made with your framing. I had some ideas that were hard to categorize cleanly into epistemics, coordination, or direct risk targeting. 

I included a few more ideas for areas where AI tools, marginal automated research, and cognitive abundance might be well applied. I was going for a similar vibe, so I'm sorry if I overlap a lot. I will try to only mention things you didn't explicitly suggest:

Epistemics:

• you mention bench-marking as a strategy for accelerating specific AI applications, but it also deserves mention as an epistemic tool.  METR style elicitation tests too
  • I should say up front that I don't know if, due to acceleration and iteration effects you mention, eg. FrontierMath and lastexam.ai are explicitly "race-dynamic-accelerating in a way that overshadows their epistemic usefulness; even METR's task horizon metric could be accused here.
  • From a certain perspective, I would consider benchmarks and METR elicitation tests natural compliments to mech interp and AI capabilities forecasting
  • this would include capabilities and threats assessment (hopefully we can actively iterate downwards on risk assessment scores)
• broad economics and societal impact research
  • the effects of having AI more or less "do the economy" seem vast and differentially accelerating understanding and strategy there seems like a non-trivial application relevant to the long term future of humanity
  • wealth inequality and the looming threat of mass unemployment (at minimum, this seems important for instability and coordination reasons even if one were too utilitarian / long termist to care for normal reasons)
• I think it would be good to accelerate "Risk evaluation" in a sense that I think was defined really elegantly by Joe Carlsmith in "Paths and waystations in AI safety" ^[1]
  • building naturally from there, forecasting systems could be specifically applied to DAID and DPD; I know this is a little "ouroboros" to suggest but I think it works

Coordination-enabling:

• movement building research and macro-strategy, AI-fueled activism, political coalition building, AI research into and tools for strengthening democracy
• auto research into deliberative mini-publics, improved voting systems (eg. ranked choice, liquid democracy, quadratic voting, anti-gerrymandering solutions), secure digital voting platforms, improved checks and balances (eg. strong multi-stakeholder oversight, whistleblower protections, human rights), non-censorship oriented solutions to misinformation
   

Risk-targeting:

• I know it is not the main thrust of "existential security", but I think it worth considering the potential for "abundant cognition" to welfare / sentience research (eg. bio and AI). This seems really important from a lot of perspectives, for a lot of reasons:

  • AI Safety might be worse if the AIs are "discontent"
  • we could lock in a future where most people are suffering terribly which would not count as existential security
  • it seems worthwhile to know if the AI workers are suffering ASAP for normal "avoid doing moral catastrophes" reasons
  • we could unlock huge amounts of welfare or learn to avoid huge amount of pain; (cf. "hedonium" or the Far Out Initiative)

  That said, I have not really considered the offense / defense balance here. We may discover how to simulate suffering for much cheaper than pleasure or something horrendous like that. Or there might be info hazards. This space seems so high stakes and hard to chart.

Some mix:

• Certain forms of monitoring and openly researching other people's actions seem like a mix of epistemics and coordination. For example, I had listed some stuff about ie. AI for broadly OSINT-based investigative journalism, AI lab watch, legislator scorecards, and similar. These are kind of information for the sake of coordination. 

I know I included some moonshots. This all depends on what AI systems we are talking about and what they are actually helpful with I guess. I would hate for EA to bet too hard on any of this stuff and accidentally flood the zone of key areas with LLM "slop" or whatever.

Also, to state the obvious, there may be some risk of correlated exposure if you pin too much of your existential security with the crucial aid of unreliable, untrustworthy AIs. Maybe Hal 9000 isn't always the entity to trust with your most critical security. 

Lots to think about here! Thanks! 

1. ^{^}

   Joe Carlsmith: "Risk evaluation tracks the safety range and the capability frontier, and it forecasts where a given form of AI development/deployment will put them.

   • Paradigm examples include:
     • evals for dangerous capabilities and motivations;
     • forecasts about where a given sort of development/deployment will lead (e.g., via scaling laws, expert assessments, attempts to apply human and/or AI forecasting to relevant questions, etc);
     • general improvements to our scientific understanding of AI
     • structured safety cases and/or cost-benefit analyses that draw on this information."

I have a few questions about the space of EA communities.

You mention 

Projects that build communities focused on impartial, scope-sensitive and ambitious altruism.

as in scope. I am curious what existing examples you have of communities that place emphasis on these values aside from the core "EA" brand? 

I know that GWWC kind of exists as it's own community independent of "EA" to ~some extent, but honestly I am unclear to what extent. Also, I guess LessWrong and the broader rationality-cinematic-universe might kind of fit here too, but realistically whenever scope sensitive altruism is the topic of discussion on LessWrong an EA Forum cross-post is likely. Are there any big "impartial, scope-sensitive and ambitious altruism" communities I am missing? I know there are several non-profits independently working on charity evaluation and that sort of thing, but I am not very aware of distinct "communities" per say?

Some of my motivation for asking is that I actually think there is a lot of potential when it comes to EA-esque communities that aren't actually officially "EA" or "EA Groups". In particular, I am personally interested in the idea of local EA-esque community groups with a more proactive focus on fellowship, loving community, social kindness/fraternity, and providing people a context for profound/meaningful experiences. Still championing many EA-values (scope-sensitivity, broad moral circles, proactive ethics) and EA tools (effective giving, research oriented, and ethics-driven careers), but in the context of a group which is a shade or two more like churches, humanist associations, and the Sunday Assembly and a shade or two less like Rotary Clubs or professional groups. 

That's just one idea, but I'm really trying to ask about the broader status of EA-diaspora communities / non-canonically "EA" community groups under EAIF? I would like to more clearly understand what the canonical "stewards of the EA brand" in CEA and the EAIF have in mind for the future of EA groups and the movement as a whole? What does success look like here; what are these groups trying to be / blossom into? And to the extent that my personal vision for "the future of EA" is different, is a clear-break / diaspora the way to go?

Thanks!
 

I've seen EA meditation, EA bouldering, EA clubbing, EA whatever. Orgs seem to want everyone and the janitor to be "aligned". Everyone's dating each other. It seems that we're even afraid of them.

I am not in the Bay Area or London, so I guess I'm maybe not personally familiar with the full extent of what you're describing, but there are elements of this that sound mostly positive to me. 

Like, of course, it is possible to overemphasize the importance of culture fit and mission alignment when making hiring decisions. It seems like a balance and depends on the circumstance and I don't have much to say there. 

As far as the extensive EA fraternizing goes, that actually seems mostly good. Like, to the extent that EA is a "community", it doesn't seem surprising or bad that people are drawn to hang out. Church groups do that sort of thing all the time for example. People often like hanging out with others with shared values, interests, experiences, outlook, and cultural touchstones. Granted, there are healthy and unhealthy forms of this. 

I'm sure there's potential for things to get inappropriate and for inappropriate power dynamics to occur when it comes to ambiguous overlap between professional contexts, personal relationships, and shared social circles. At their best though, social communities can provide people a lot of value and support. 

Why is "EA clubbing" a bad thing?