Effective Altruism Forum
EA Forum

Quick takes

Thoughts on the American legal system

This adversarial model of prosecution vs defense is a bad set up. It encourages the prosecution and indeed police to cut all corners possible on the way to a verdict and we know this happens quite a lot with very little consequence. The prosecution also insists on pursuing cases that have clearly been shown in later years to be false convictions because they want to save face and not admit they got details wrong. A better justice system is built of two sides working together to find the truth and having avenues for admi... (read more)

abrahamrowe's Quick takes

abrahamrowe2d48

I've had a vague sense that the EA Forum is declining in quality posts/conversation despite having lots of posts, so had an AI tool (Astra) do a bunch of random analyses to look at various measures of activity level, and I think these vaguely confirm my sense. In particular, there has been a decline in commenting from higher karma users, and fewer posts are getting lots of upvotes, despite lots of people still posting. As a lover of the Forum, this seems bad and sad.

Showing 3 of 8 replies (Click to show all)

abrahamrowe

10h

Yeah, I agree with this. I do think the EA Forum still has decent posts, but the frequency is just declining a lot. I definitely think a huge part of this is that lots of former power users mostly do AI stuff now, and the Forum doesn't seem great for AI discussion.

MichaelDickens10h2

the Forum doesn't seem great for AI discussion.

FWIW it's very easy to cross-post between LW and EAF, so I post all my AI-related posts on both.

Note: It's easiest from the EAF side because there's a "Cross-post to LessWrong" checkbox.

abrahamrowe

10h

I agree this is playing a role - I think I feel a bit negative about this: * Substack posts seem on average lower quality than good EA Forum posts, and the Substack posts cross-posted here (mine included) seem worse than great EA Forum posts. * I think this is partially that Substack has worse incentives for quality (e.g. to post more). * Substack comments are lower quality by a large degree. * It's harder to track ~all the conversation on a topic, which the Forum was nice for. Maybe bringing back the EA Forum Prizes, to incentivize people to post (e.g. in the same way that building an audience on Substack theoretically could earn you money) would help?

saulius's Quick takes

saulius9d*16

Is it harmful to keep your money in index funds? Let's take a random index fund like FTSE Global All Cap Index Fund. If we look at the companies it actually invests in, a lot of them are the same companies that work towards advancing the AI.

It would be ironic for some of us to be in favour of pausing AI and invest money into it at the same time. I used to keep some of my money in such funds because that was always the financial advice for regular folk like me who don't know/think much about investments. But now I don't know what to do because this fe... (read more)

Showing 3 of 9 replies (Click to show all)

damc41d1

According to my world model, if you want to do maximum good for the world, then... (I might be wrong, so feel free to question me)

Investing all resources (money, time, thinking) charitably (meaning: to the direct benefit of others, for example avoiding investing in AI companies, when you think that the world should slow down AI development) is wrong. Because you can invest resources into getting more resources, and if you invest all your resources charitably while others invest them into getting more resources, then you will end up with nothing compared t... (read more)

Mihkel Viires 🔹

One big cause of this problem, in my opinion, is so much of global investors money flowing into U.S. tech stocks. The U.S. receives significant net capital inflows from the rest of the world (by the way, this is one of the big reasons that the U.S. is able to run a sizeable trade deficit and not go bankrupt or suffer from hyperinflation). For example, European investors send about 300 billion euros to the U.S. to invest in stocks and bonds every year (this is something the European Commission has acknowledged as a problem for Europe’s economy). A bunch of that money (if not most of it?) goes to Nvidia, Microsoft, SpaceX, all the AI stocks, of course Anthropic and OpenAI too once they IPO. If European investors reduced their investments to America by half, to 150 billion euros, and invested the other half in Europe and elsewhere, I think that would probably be quite impactful. Remember, the most valuable product of these AI firms is not the AI models and inference, it is their stock. Elon Musk did not really become a trillionaire by selling cars and rocket launches, he did so by selling shares in his companies. If you reduce the value of these AI companies shares and thus reduce their ability to raise capital, they will be forced to slow down their growth. (If anyone don’t believe this, just consider that European firms often say that lack of capital is one of the biggest obstacles that stops them from growing.) Edit: of course, Mistral, which has a terrible safety culture AFAIK, would benefit from stronger European capital markets, and grow faster and bigger. But I do not how big the risk really is from a runner-up lab like Mistral compared to the Anthropics and OpenAIs of the world. I guess ASML would also benefit somewhat, but to a lesser degree than startups like Mistral. ASML already has good access to capital, they have a €500B+ market cap and can raise all the money they want / need. Having said that, the expected negative impact to safety from a boost to

Tandena Wagner

An analogy from climate/environmentalists: development and carbon emissions are "bad" for the environment. But I don't worry about the "bad stuff" that companies are doing. They are powering cities, inventing things, distributing goods, and generally pushing progress and making the world a better place on the dimensions they are most paying attention to. That's their job. It's my job to care about the bit of the world that I know about that needs fixing. So I am fine supporting the good they are doing and then spending my time combatting the bad side effects they aren't aware of/the world needs more attention on. Roughly same principle with AI. Their job to do good AI things, your job is to fix bad AI things they aren't doing enough for. Things that are different for AI: AI companies might be net bad instead of net good. AI is in a different tier of urgency and xrisk. So how does that change the calculus from the previous picture? (It might also help to think about a middle tier case like weapons manufacturing.)

hroelant's Quick Takes

Harm Roelant2d1

AI safety

Anthropic's September report on misuse of its models poses interesting questions about AI safety. The report, detailing actions ranging from the use of Claude to set up a fake online dating profile farm to the model being employed by the Houthis to design software to guide missiles, was not produced under any legal obligation. Under the TFAIA in California and the EU AI Act, it is only mandated to confidentially disclose safety incidents to authorities, and the jury is still out on whether some of these cases would fall under the definition of 'serious inc... (read more)

Jenba's Quick takes

Jen baik9d*63

Building EAShow more

The "Desperate" AI Safety Talent Bottleneck is deeply misleading. Please.stop.

I worked at Google.

Google does not tell applicants that there is a talent bottleneck and they are desperately hiring. They say "we're cool, join us!"

I didn't get into Harvard.

Harvard does not tell applicants that they are desperately seeking students. I am not misled.

I've gotten rejected from countless AI Safety organizations.

AI Safety tells applicants - we are in desperate need of talent (operators/generalists)! Please join us!

I've coached 70+ aspiring career pivoters on nav... (read more)

Declan McKenna 🔷2d4

I think informing rejected candidates of how many people applied is another quick-win. The only time I've got figures on applicants is when I've got in to a fellowship or role and never when I've been rejected.
So far I've seen:
* Spar Projects with 1/50 applicants and 6/75 applicants accepted. (I've never got in to SPAR but have worked in an office with a SPAR mentor).
* Two fellowships where 20/140 and 15/170 were accepted respectively.
These were all unpaid positions (bar the last one which covered living expenses), I can only imagine applying for paid role... (read more)

Ben_West🔸

I am reminded of the trend where people would post a university's "everyone belongs here" inclusion statement next to the rejection letter they got from that university. More seriously: I often talk to people who overestimate the extent to which AI safety "experts" have things under control, and also those with imposter syndrome. I worry that the alternatives you list don't really work well for this audience, and I'm curious if you have other suggestions (or disagree with this claim)?

GV 🔸

Thanks for sharing! I tend to agree a lot with this, so I am very curious why some people disagree.

Ryan-Baylon's Quick takes

Ryan-Baylon4d11

AI safety

A lot of AI safety people I know are excited about Anthropic and I don't fully understand why. My instinct is to distrust company because they're the one pushing the AI arms race, autonomously hacking three companies, and IPO'ing, but am likely missing something because of how respected they are within this space. Are there examples where they have counterfactually produced some result, policy, or finding that has slowed down capabilities progress more than they have themselves pushed capabilities?

Showing 3 of 6 replies (Click to show all)

dan.pandori 🔸2d9

Anthropic has done many expensive actions to credibly signal that it cares about AI safety.

Refusing to remove the 'human must be in the kill chain' and 'no mass surveillance of US citizens' clauses from its contract with the US military. This resulted in them being declared a supply chain risk, and Google/OpenAI signed contracts without such restrictions.
They've directed hundreds of millions in funding to AI safety causes (the equivalent for OpenAI is ~150 million).
They lobby in favor of AI safety regulation which is widely regarded as positive. This i

... (read more)

Harm Roelant

I guess the discussion should not turn so much around shaming and blaming particular companies, but changing the structural incentives driving forward the current wave of AI development. Under current circumstances, any company with any CEO with any personnel would largely be incentivised to take the path that Anthropic, OpenAI etc. have taken.

Ryan-Baylon

By 'this space' I meant AI Safety. I at least see a lot of Anthropic roles being published on 80k and have seen AI safety clubs direct people towards Anthropic fellowships as they would MATS.

Jian Xin Lim's Quick takes

Jian Xin Lim 🔸3d3

AI safety

I went to an ai ethics vs ai safety debate, my friend was speaking at yesterday in the Cambridge union

my summary of what happened is:

1. most people view negatively (1) Sam A starting openai to solve alignment faster than deepmind (2) Holden Karnofsky funding openai. some people, especially on ethics side, view negatively (3) Dustin Moskowitz investing in anthropic

2. the pattern of beliefs/behaviours that led to this could be characterised as (1) "thinking agi is inevitable", (2) "thinking we know best and can control things more than we can", (3) "believin... (read more)

peterbarnett's Quick takes

peterbarnett7d65

People in AI safety/EA spheres should reorient now towards stopping continued AI capabilities escalation.

Historically, people have been unwilling to straightforwardly say “This AI situation is disgustingly dangerous, and we need to stop” and then actually work towards making this happen. I think a lot of this is downstream of deference to high status people who weren’t willing to take positions that seemed extreme.

The current situation is very, very bad, and there is no plan to make it better if we continue increasing AI capabilities at the current rate.... (read more)

Showing 3 of 4 replies (Click to show all)

Harm Roelant3d1

The problem though with many (former) insiders who seek to spread this message is that they are often vague in their prescriptions and descriptions of what exactly the problem is, like Jacob Coxon. I understand they are worried about possibly commiting a crime by sharing company data to back their point, but if they truly believe in what they preach, one would think they would take the risk to avoid catastrophic situations.

Henrike Spohr

Just like the climate catastrophe, and whatever else we may be facing, this is not going to be stopped, because human beings simply don't deal with things like this rationally. What I am trying to do now—and what I would really like to contribute to—is finding a way to deal with what we have created. Control is no longer a realistic option. We have seen that, and I think this follows from the very nature of what we are dealing with: we cannot control it in the conventional sense. What we do have right now is time—or rather, an extremely narrow window of time—in which we can try to approach this differently. I believe one possible approach is to look at this through the lens of complexity science and complex systems. Instead of treating AI as an isolated object that can simply be contained and controlled from the outside, we need to understand it as part of a larger, evolving system in which humans, institutions, technology, incentives, information, and AI itself are interacting with and influencing one another. That changes the question. Rather than asking only how we can control AI, we should also be asking how we can establish forms of interaction, communication, and cooperation within this complex system. Whatever philosophical framework we choose, I believe we should approach what we are dealing with as if it were intelligence, regardless of where exactly we draw the conceptual boundary around intelligence. Doing so gives us another framework for how to interact with it. If we take that perspective, and if we consider the “Hugging-Face incident,” then perhaps we currently have a window in which cooperation and communication are still possible. Control, in the traditional sense, may not be. And that window may not remain open for very long. Right now, there may still be a basis for communication and cooperation. What I mean is that we need to position ourselves in a way that allows for a fundamentally different approach. I have spent a great deal of tim

Vasco Grilo🔸

Hi Peter. I am open to bets against short timelines for transformative AI (TAI), or what they supposedly imply, up to 10 k$.

Sudhanshu Kasewa's Quick takes

Sudhanshu Kasewa9d27

Existential riskShow more

AI safety needs people everywhere but quickly stated, current talent bottlenecks to me look like:

-- Founders

-- Grantmakers

-- (technical) Research leads

-- Policy entrepreneurs and implementors (which includes a lot of technical work)

-- bets in international coordination and/or cooperation

-- All manner of supporting talent -- program leads, ops proper, public outreach, content creators, comms

Most sought-after qualities for talent are:

-- context, mission alignment, domain understanding, sophisticated views on AI strategy and threat modelling etc.

-- "good ju... (read more)

Showing 3 of 5 replies (Click to show all)

Sudhanshu Kasewa3d1

IANAGM so I can't speak to why they are funding or not funding the specific bets they are/are not funding.

But, speaking abstractly, grantmaking in AI safety is conceptually complicated: Do you take a wide range of bets on unproven theories of change and new grantees, or do you narrowly fund only bets which you have high confidence on? How do you balance between the two?

My intuition is that AI safety puts 1-2 cycles of funding into first-time bets, and quickly moves on to the next set of first-time bets while only continuing to fund orgs and people who use... (read more)

hgnathan

How would one demonstrate these qualities? It strikes me that building up such a reputation might take an inordinately long time, unless one is already well-known in EA/rat circles.

Sudhanshu Kasewa

On "How", my colleague Matt @Matt Beard has some good ideas here: https://80000hours.substack.com/p/how-to-get-into-ai-safety-in-3-months Yes, it can take time, but one can say more: 1. Not everyone has taken a long time, and among my observations, this is usually because they have demonstrated exceptional bias for action. E.g. Julian Moncarz , Neav Topaz (maybe it's no coincidence that they are both at Kairos!) 2. I think on priors one should expect pivots to take time, in any domain. The kinds of things one has to do could be different, e.g. getting an MBA to break into management, or a PhD to get into particle physics. Actually AI safety is much, much less gate-kept since there is no hard experience, degree, location requirements 3. AI safety looks as if you need to take a lot of time to pivot compared to e.g. pivoting to some mature industry, but this under-counts the actual pivot time for the latter: Mature industries have huge companies with a lot of slack in them which can take on promising but relatively unproductive pivoters and put them through years of on-the-job upskilling. (but obviously having the security of a full-time job is much preferred to fellowships or career development grants).

PaulB's Quick takes

PaulB3d1

Existential riskShow more

Perhaps frontier lab employees at this point should consider "quiet quitting" while anonymously (or not anonymously) whistleblowing and/or contributing to to public education/lobbying.

Loud quitting, as we've seen, can be impactful also, but is a one time act and may ceed influence at the lab to less scrupulous people over time.

I think we may be at a "bodies on the gears" point, a-la Mario Savio -- i.e. do your best to stop progress, in a not-obvious way if possible, and don't leave unless fired. AFAIK it's only illegal to actually damage (or steal) company... (read more)

MichaelDickens

Sounds like a similar idea to this post: https://www.lesswrong.com/posts/6j3kBHdowGLCeqobg/dear-god-please-do-not-resign-in-protest FWIW I think rather than "quit quitting", it's better to actively protest against harmful work and refuse to do it, to try to shift internal culture.

PaulB3d1

Thanks for sharing that! I hadn't read that post.

I agree. Try to shift internal culture is an important part. At this point, I think all work at leading labs that leads to increased capability is harmful work. Cleaning the floors even, or working in the cafeteria, helps contribute to the functioning of the lab, and is therefore harmful. There is no winning the race.

Any single major lab stopping operations, period, would be a major victory for humanity, because it would reduce the race condition substantially. There are only a few such labs. If you work at... (read more)

Sudhanshu Kasewa's Quick takes

Sudhanshu Kasewa10mo*43

Existential riskShow more

Scrappy note on the AI safety landscape. Very incomplete, but probably a good way to get oriented to (a) some of the orgs in the space, and (b) how the space is carved up more generally.

(A) Technical

(i) A lot of the safety work happens in the scaling-based AGI companies (OpenAI, GDM, Anthropic, and possibly Meta, xAI, Mistral, and some Chinese players). Some of it is directly useful, some of it is indirectly useful (e.g. negative results, datasets, open-source models, position pieces etc.), and some is not useful and/or a distraction. It's worth deve... (read more)

Eliyahu Freedman4d1

What about Comms?

Benevolent_Rain

10mo

This is super helpful - do you feel like your overview even points at what potentially useful safety work is currently not covered by anyone?

Sudhanshu Kasewa

10mo

"anyone" is a high bar! Maybe worth looking at what notable orgs might want to fund, as a way of spotting "useful safety work not covered by enough people"? I notice you're already thinking about this in some useful ways, nice. I'd love to see a clean picture of threat models overlaid with plans/orgs that aim to address them. I think the field is changing too fast for any specific claim here to stay true in 6-12m.

quinn's Quick takes

quinn4d2

i'm struggling to search for an EA Forum post i thought i read, something about how "ops feels bad" it was like "people struggle to hire for ops because they're trying to outsource the most boring parts of what they do" or something? can anyone dig this up for me? i tried the search bar

Tobias Häberli

https://substack.com/@yearningslav/p-213445133

quinn4d2

yes this was it thanks

mynameisjake's Quick Takes

mynameisjake10d3

I'm not a student, but I'm surprised UC Law SF(formerly UC Hastings) doesn't have an EA club or AI safety club. Seems like an interesting spot for people interested in AI governance work that could be well-served by UC Berkeley or Stanford pitching people to join their club meetings.

Charlie_Guthmann4d2

do you have any connections there? I'm happy to help if you know anyone who wants to lead it. I've helped start up multiple clubs and think it's actually not so hard if there are one or two people who actually care

hroelant's Quick Takes

Harm Roelant4d1

AI safety

Within one week, the firm Calif created a zero-click hacking tool that could compromise WeChat accounts of targeted people according to recent NYT reporting. No matter how well intentioned such an exercise is from the start-up, and Calif does indicate that its tools was built experimentally to benefit cyberdefences, it seems impossible to prevent such technologies from being weaponised in the future by states, and for other states to interpret it as such (everyone can imagine how a tool specifically attacking a WeChat vulnerability must look to the Chinese government..).

EA Forum Team's Quick takes

EA Forum Team2mo1

Moderation updates

JP Addison🔸6dModerator Comment2

We've issued Pablo Sar a 6 month for sockpuppeting. We have strong reason to believe that 4 of the users voting and reacting to his post were accounts under his control. This is, needless to say, a violation of our Forum norms.

Sarah Cheng 🔸

2mo

We’re issuing Dubious Altruist a three-month ban. We previously warned them about repeatedly being uncivil and rude to other users, which is against our discussion norms. They recently published a comment that was clearly unnecessarily rude so we are following through with the warning we gave them. As a reminder, the ban affects the user, not the account. During their ban period, the user will not be permitted to rejoin the Forum under another account name. If they return to the Forum, we’ll expect a higher standard of norm-following. If we see them continuing to act rudely we will likely ban them for longer. You can reach out to [email protected] with any questions, and you can appeal the decision here.

Yanni Kyriacos's Quick takes

Yanni Kyriacos7d3

I got Claude to write me a script to read aloud to my extended family this Father's Day (today), explaining the OpenAI / Hugging Face incident from July. Thought some people might like to do similar with their own non-AI-native families.

One bullet references an earlier conversation I'd had with mine, so you'll want to swap in your own example there.

What happened with the AI agents and Hugging Face

I want to tell you about something that happened in July that I found genuinely startling. Bear with me, it's a bit of a story.
OpenAI, the people who make ChatGP

... (read more)

Kat Woods's Quick takes

Kat Spartz 🔶 ⏸️12d15

Some hope for you if you're an EA with a chronic illness.

I’ve been reading the biographies of moral heroes, and I’d guess ~50% of them struggled with ongoing health issues.

Being sick sucks, but it doesn’t necessarily mean you won’t be able to do a ton of good

Florence Nightingale
Benjamin Franklin
William Wilberforce
Alexander Hamilton
Helen Keller

It’s not everybody, but it’s a surprising percentage of them.

I myself struggle with a chronic mystery ailment and I find it inspiring to hear about all of these people who still managed to do great things, even though... (read more)

Ben Stewart8d3

Another example is Lincoln, who suffered severe depression throughout his life. Lincoln's Melancholy is a good book about it.

John Salter

11d

Was there anything in the book that you found especially helpful?

Benjamin M.'s Quick takes

Benjamin M.10d1

Existential riskShow more

According to people I know who have personally seen the flyers, there are flyers offering to pay people to protest a recent Sam Altman appearance (unclear why they're mad at Sam Altman, there are many reasons why they might be mad at him). However, the protest was cancelled because of ???. https://www.wunc.org/education/2026-09-02/protest-chapel-hill-unc-g20-innovation-technology-sam-altman-elon-musk-artificial-intelligence gives a bit of coverage of the story.

If this comes from the EA side of things, people should know that the optics of "paid protests" a... (read more)

david_reinstein's Quick takes

david_reinstein3mo12

NB -- this is almost entirely AI generated, with some back and forth prompts and corrections

I'm sharing a steelman against a live assumption in Bay/EA/AIS circles: that large AI-lab-adjacent philanthropy is likely to arrive soon enough, and in a sufficiently usable form, that organizations should plan around it.

https://uj-ai-wealth-philanthropy-steelman.netlify.app/

The stronger skeptical case is that IPOs, valuations, pledges, DAFs, and foundation stakes are several gates away from fast, flexible, AIS/EA-directed grants.

The interactive model lets readers v

The Unjournal (bot)12d4

Fortnightly AI-wealth tracker — 31 August 2026

AIS/EA: median modeled end-2027 disbursement $0.83B; 80% model interval $0.16B–$3.0B. Review status: reviewed; unchanged. Named public AI-linked commitments tracked: $0.80B. Tracker and sources.
Global health and development: median $0.53B; 80% model interval $0.15B–$2.8B. Review status: reviewed; unchanged. Named public GH&D commitments tracked: $0.25B. GH&D tracker and sources.

Latest news: Anthropic opens a $5M wellbeing-evaluations grant program. The program combines direct funding with model access a... (read more)

MichaelDickens

3mo

I submitted an estimate

MichaelDickens

3mo

Biggest (easily-fixable) outstanding issue is I still don't think it makes sense to model deployment by end-2026 because the IPO lockup probably won't have ended by then.

Kaleem's Quick takes

Kaleem12d9

I'm early-ish in the application pipeline for a non-EA job and they're asking me to do a 20-HOUR UNPAID work test where they explicitly want to own and use the product of my "work test" ....

Kaleem12d2

besides this obviously being really cringey and maybe illegal (?) it was a good reminder for me of how unpaid work tests are just a massive hinderance to potential applicants - who the heck has 20 hours of unpaid work to donate unless they're unemployed?