otoh I've found watching "famous" people debate helpful for taking them off the pedestal. It's demystifying to watch them argue things and think about things in public rather than just reading their more polished thoughts. The former almost always makes impressive-looking people seem less impressive.  

That sounds like it would be helpful, but I would also want people to have a healthier relationship with having an impact and with intelligence than I see some EAs having. It's also okay to not be the type of person who would be good at the types of jobs that EAs currently think are most important or would be most important for "saving the world". There's more to life than that. 

I'm also curious about the answer to this question. For people I know in that category (which disincludes anyone who just stopped engaging with AI safety or EA entirely), many are working as software engineers or are on short-term grants to skill up. I'd expect more of them to do ML engineering if there were more jobs in that relative to more general software engineering. A couple of people I know after getting rejected from AI safety-relevant jobs or opportunities have also made the decision to do master's degrees or PhDs with the expectation that that might help, which is an option that's more available to people who are younger. 

I will probably be publishing a post on my best guesses for how public discourse and interest in AI existential risk over the past few months should update EA's priorities: what things seem less useful now, what things seem more useful, what things were surprising to me about the recent public interest that I suspect are also surprising to others. I will be writing this post as an EA and AI safety random, with the expectation that others who are more knowledgeable will tell me where they think I'm wrong.  

I mostly haven't been thinking about what the ideal effective altruism community would look like, because it seems like most of the value of effective altruism might just get approximated to what impact it has on steering the world towards better AGI futures. But I think even in worlds where AI risk wasn't a problem, the effective altruism movement seems lackluster in some ways.

I am thinking especially of the effect that it often has on university students and younger people. My sense is that EA sometimes influences those people to be closed-minded or at least doesn't contribute to making them as ambitious or interested in exploring things outside "conventional EA" as I think would be ideal. Students who come across EA often become too attached to specific EA organisations or paths to impact suggested by existing EA institutions. 

In an EA community that was more ambitiously impactful, there would be a higher proportion of folks at least strongly considering doing things like starting startups that could be really big, traveling to various parts of the world to form a view about how poverty affects welfare, having long google docs with their current best guesses for how to get rid of factory farming, looking at non-"EA" sources to figure out what more effective interventions GiveWell might be missing perhaps because they're somewhat controversial, doing more effective science/medical research, writing something on the topic of better thinking and decision-making that could be as influential as Eliezer's sequences, expressing curiosity about the question of whether charity is even the best way to improve human welfare, trying to fix science. 

And a lower proportion of these folks would be applying to jobs on the 80,000 Hours job board or choosing to spend more time within the EA community rather than interacting with the most ambitious, intelligent, and interesting people amongst their general peers. 

I found the conversations I had with some early-career AI safety enthusiasts to show a lack of understanding of paths to x-risk and criticisms of key assumptions. I’m wondering if the early-stage AI field-building funnel might cause an echo chamber of unexamined AI panic that undermines general epistemics and cause-neutral principles.

I think people new to EA not knowing a lot about specific cause areas they're excited about isn't more true for AI x-risk than other cause areas. For example, I suspect if you asked animal welfare or global health enthusiasts who are as new as the folks into AI safety you talked to about the key assumptions relating to different animal welfare or global health interventions, you'd get similar results. It just seems to matter more for AI x-risk though since having an impact there relies more strongly on having better models. 

+1, also interested

Thanks! Does that depend on the empirical question of how costly it would be for the AI to protect us and how much the aliens care about us or is the first number too small that there's almost always going to be someone willing to trade?

I imagine the civilisations that care about intelligent life far away have lots of others they'd want to pay to protect. Also unsure about what form their "protect Earth life" preference takes - if it is conservationist style "preserve Earth in its current form forever" then that also sounds bad because I think Earth right now might be net negative due to animal suffering. Though hopefully there not being sentient beings that suffer is a common enough preference in the universe. And that there are more aliens who would make reasonable-to-us tradeoffs with suffering such that we don't end up dying due to particularly suffering focused aliens.

I don't think it makes any arguments? I also expect less to be convinced that factory-farmed animals have net positive lives, that wild animals might seems easier to defend