This is a special post for quick takes by Ethan Beri. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Comment Permalink
I've recently been writing a long-form post, and I realised that it's taking a while. I was sort of struck by the thought: is everyone doing this? When I see people talk about spending too much time on the forum (and I usually don't - I think I've only seen two or three people say this), it's usually to do with doom scrolling, or something like that. But might people also be spending a lot of time just writing stuff? I'm not sure of the concrete opportunity cost here, but I'm sure there's some. I'm not especially well-versed in the "meta trap" thing - I think this was a debate people had before I got interested in EA - but it seems like this is way it could (or does!) happen. Thoughts?
I'm not really too concerned with quality - I'm just much more worried about time. Actually, I think a lot of stuff on EA Forum/LW is really good, and I've learned a lot here. It just seems like an awful lot of people write an awful lot of stuff, and I'm not really sure everyone needs to spend so long writing. That said, I'm not sure how you'd fix this other than implementing something like the existing quick takes feature.
Hey - I’d be really keen to hear peoples' thoughts on the following career/education decision I'm considering (esp. people who think about AI a lot):
- I’m about to start my undergrad studying PPE at Oxford.
- I’m wondering whether re-applying this year to study CS & philosophy at Oxford (while doing my PPE degree) is a good idea.
- This doesn’t mean I have to quit PPE or anything.
- I’d also have to start CS & philosophy from scratch the following year.
- My current thinking is that I shouldn’t do this - I think it’s unlikely that I’ll be sufficiently good to, say, get into a top 10 ML PhD or anything, so the technical knowledge that I’d need for the AI-related paths I’m considering (policy, research, journalism, maybe software engineering) is either pretty limited (the first three options) or much easier to self-teach and less reliant on credentials (software engineering).
- I should also add that I’m currently okay at programming anyway, and plan to develop this alongside my degree regardless of what I do - it seems like a broadly useful skill that’ll also give me more optionality.
- I do have a suspicion that I’m being self-limiting re the PhD thing - if everyone else is starting from a (relatively) blank slate, maybe I’d be on equal footing?
- That said, I also have my suspicions that the PhD route is actually my highest-impact option: I’m stuck between 1) deferring to 80K here, and 2) my other feeling that enacting policy/doing policy research might be higher-impact/more tractable.
- They’re also obviously super competitive, and seem to only be getting more so.
- One major uncertainty I have is whether, for things like policy, a PPE degree (or anything politics-y/economics-y) really matters. I’m a UK citizen, and given the record of UK politicians who did PPE at Oxford, it seems like it might?
What mistakes am I making here/am I being too self-limiting? I should add that (from talking to people at Oxford) I’ll have quite a lot of time to study other stuff on the side during my PPE degree. Thanks for reading this, if you’ve got this far! I’d greatly appreciate any comments.
Nudge to seriously consider applying for 80,000 hours personal advising if you haven't already: https://80000hours.org/speak-with-us/
My guess is they'd be able to help you think this through!
Hey, tough choice! Personally I’d lean towards PPE. Primarily that’s driven by the high opportunity cost of another year in school. Which major you choose seems less important than finding something you love and doing good work in it a year sooner.
Two other factors: First, you can learn AI outside of the classroom fairly well, especially since you can already program. I’m an economics major who’s taken a few classes in CS and done a lot of self-study, and that’s been enough to work on some AI research projects. Second, policy is plausibly more important for AI safety than technical research. There’s been a lot of government focus on slowing down AI progress lately, while technical safety research seems like it will need more time to prepare for advanced AI. The fact that you won’t graduate for a few years mitigates this a bit — maybe priorities will have changed by the time you graduate.
What would you do during a year off? Is it studying PPE for one year? I think a lot of the value of education comes from signaling, so without a diploma to show for it this year of PPE might not be worth much. If there’s a job or scholarship or something, that might be more compelling. Some people would suggest self-study, but I’ve spent time working on my own projects at home, and personally I found it much less motivating and educational than being in school or working.
Those are just my quick impressions, don’t lean too much on anyone (including 80K!). You have to understand the motivations for a plan for yourself in order to execute it well. Good luck, always happy to chat about stuff.
Hey, thanks for your comment! I hadn't really realised the extent to which someone can study full-time while also skilling up in research engineering - that definitely makes me feel more willing to go for PPE.
Re your third paragraph, I wouldn't have a year off - it'd just be like doing a year of PPE, followed by three years of CS & philosophy. I do have a scholarship, and would do the first year of PPE anyway in case I didn't get into CS & phil.
Either way, your first point does point me more in the direction of just sticking with PPE :)
Would an AI governance book that covered the present landscape of gov-related topics (maybe like a book version of the FHI's AI Governance Research Agenda?) be useful?
We're currently at a weird point where there's a lot of interest in AI - news coverage, investment, etc. It feels weird to not be trying to shape the conversation on AI risk more than we are now. I'm well aware that this sort of thing can backfire, and I'm aware that most people are highly sceptical of trying not to "politicise" issues like these, but it might be a good idea.
If it was written by, say, Toby Ord - or anyone sufficiently detached from American left/right politics, with enough prestige, background, and experience with writing books like these - I feel like it might be really valuable.
It might also be more approachable than other books covering AI risk, like, say, Superintelligence. It might also seem a little more concrete, because it might cover scenarios that are easier for most people to imagine/scenarios that are more near-term, and less "sci-fi".
Thoughts on this?
In my understanding, going from manuscript completion to publication probably takes 1-2 years. This is long enough that new developments in AI capabilities/regulations/treaties would come about, but worse, AI governance is a fast growing academic field right now. I imagine that the state-of-the-art in AI gov research/analysis frameworks could look quite different in a couple of years.
Curated and popular this week
·
In recent months, the CEOs of leading AI companies have grown increasingly confident about rapid progress:
* OpenAI's Sam Altman: Shifted from saying in November "the rate of progress continues" to declaring in January "we are now confident we know how to build AGI"
* Anthropic's Dario Amodei: Stated in January "I'm more confident than I've ever been that we're close to powerful capabilities... in the next 2-3 years"
* Google DeepMind's Demis Hassabis: Changed from "as soon as 10 years" in autumn to "probably three to five years away" by January.
What explains the shift? Is it just hype? Or could we really have Artificial General Intelligence (AGI)[1] by 2028?
In this article, I look at what's driven recent progress, estimate how far those drivers can continue, and explain why they're likely to continue for at least four more years.
In particular, while in 2024 progress in LLM chatbots seemed to slow, a new approach started to work: teaching the models to reason using reinforcement learning.
In just a year, this let them surpass human PhDs at answering difficult scientific reasoning questions, and achieve expert-level performance on one-hour coding tasks.
We don't know how capable AGI will become, but extrapolating the recent rate of progress suggests that, by 2028, we could reach AI models with beyond-human reasoning abilities, expert-level knowledge in every domain, and that can autonomously complete multi-week projects, and progress would likely continue from there.
On this set of software engineering & computer use tasks, in 2020 AI was only able to do tasks that would typically take a human expert a couple of seconds. By 2024, that had risen to almost an hour. If the trend continues, by 2028 it'll reach several weeks.
No longer mere chatbots, these 'agent' models might soon satisfy many people's definitions of AGI — roughly, AI systems that match human performance at most knowledge work (see definition in footnote).
This means that, while the compa
·
SUMMARY:
ALLFED is launching an emergency appeal on the EA Forum due to a serious funding shortfall. Without new support, ALLFED will be forced to cut half our budget in the coming months, drastically reducing our capacity to help build global food system resilience for catastrophic scenarios like nuclear winter, a severe pandemic, or infrastructure breakdown. ALLFED is seeking $800,000 over the course of 2025 to sustain its team, continue policy-relevant research, and move forward with pilot projects that could save lives in a catastrophe. As funding priorities shift toward AI safety, we believe resilient food solutions remain a highly cost-effective way to protect the future. If you’re able to support or share this appeal, please visit allfed.info/donate.
Donate to ALLFED
FULL ARTICLE:
I (David Denkenberger) am writing alongside two of my team-mates, as ALLFED’s co-founder, to ask for your support. This is the first time in Alliance to Feed the Earth in Disaster’s (ALLFED’s) 8 year existence that we have reached out on the EA Forum with a direct funding appeal outside of Marginal Funding Week/our annual updates. I am doing so because ALLFED’s funding situation is serious, and because so much of ALLFED’s progress to date has been made possible through the support, feedback, and collaboration of the EA community.
Read our funding appeal
At ALLFED, we are deeply grateful to all our supporters, including the Survival and Flourishing Fund, which has provided the majority of our funding for years. At the end of 2024, we learned we would be receiving far less support than expected due to a shift in SFF’s strategic priorities toward AI safety.
Without additional funding, ALLFED will need to shrink. I believe the marginal cost effectiveness for improving the future and saving lives of resilience is competitive with AI Safety, even if timelines are short, because of potential AI-induced catastrophes. That is why we are asking people to donate to this emergency appeal
·
None of this article was written with AI assistance.
Introduction
There have been many, many, many attempts to lay out scenarios of AI taking over or destroying humanity. What they tend to have in common is an assumption that our doom will be sealed as a result of AI becoming significantly smarter and more powerful than the best humans, eclipsing us in skill and power and outplaying us effortlessly.
In this article, I’m going to do a twist: I’m going to write a story (and detailed analysis) about a scenario where humanity is disempowered and destroyed by AI that is dumber than us, due to a combination of hype, overconfidence, greed and anti-intellectualism. This is a scenario where instead of AI bringing untold abundance or tiling the universe with paperclips, it brings mediocrity, stagnation, and inequality.
This is not a forecast. This story probably won’t happen. But it’s a story that reflects why I am worried about AI, despite being generally dismissive of all those doom stories above. It is accompanied by an extensive, sourced list of present day issues and warning signs that are the source of my fears.
This post is divided into 3 parts:
Part 1 is my attempt at a plausible sounding science fiction story sketching out this scenario, starting with the decline of a small architecture firm and ending with nuclear Armageddon.
In part 2 I will explain, with sources, the real world current day trends that were used as ingredients for the story.
In part 3 I will analysise the likelihood of my scenario, why I think it’s very unlikely, but also why it has some clear likelihood advantages over the typical doomsday scenario.
The story of Slopworld
In the nearish future:
When the architecture firm HBM was bought out by the Vastum corporation, they announced that they would fire 99% of their architects and replace them with AI chatbots.
The architect-bot they unveiled was incredibly impressive. After uploading your site parameters, all you had to do was chat with
I'm not really too concerned with quality - I'm just much more worried about time. Actually, I think a lot of stuff on EA Forum/LW is really good, and I've learned a lot here. It just seems like an awful lot of people write an awful lot of stuff, and I'm not really sure everyone needs to spend so long writing. That said, I'm not sure how you'd fix this other than implementing something like the existing quick takes feature.