All of Buhl's Comments + Replies

Thank you! 

Worth noting that our input was also very unevenly distributed – our original idea list includes ~40% AI-related ideas, ~15% bio, ~25% movement building / community infrastructure, and only ~20% other. (this was mainly due to us having better access to AI-related project ideas via our networks). If you’re interested in pursuing biosecurity- or movement building-related projects, feel free to get in touch and I can share some of our additional ideas – for the other areas I think we don’t necessarily have great ideas.

Thanks, appreciate your comment and the compliment!

On your questions:

2. The research process does consider cost-effectiveness as a key factor – e.g., the weighted factor model we used included both an “impact potential” and a “cost” item, so projects were favoured if they had high estimated impact potential and/or a low estimated cost. “Impact potential” here means “impact with really successful (~90th percentile) execution” – we’re focusing on the extreme rather than the average case because we expect most of our expected impact to come from tail outcomes... (read more)

The quick explanation is that I don't want people to over-anchor on it, given that the inputs are extremely uncertain, and that I think that a ranked list produced by a relatively well-respected research organisation is the kind of thing people could very easily over-anchor on, even if you caveat it heavily

(I'm in a similar position to Amber: Limited background (technical or otherwise) in AI safety and just trying to make sense of things by discussing them.)

Re: "I think you need to say more about what the system is being trained for (and how we train it for that). Just saying "facts about humans are in the data" doesn't provide a causal mechanism by which the AI acts in human-like ways, any more than "facts about clouds are in the data" provides a mechanism by which the AI role-plays being a cloud."

The (main) training process for LLMs is exactly to predict h... (read more)

Thank you for the important post!
 

“we might question how well neuron counts predict overall information-processing capacity”

My naive prediction would be that many other factors predicting information-processing capacity (e.g., number of connections, conduction velocity, and refractory period) are positively correlated with neuron count, such that neuron count is pretty strongly correlated with information processing even if it only plays a minor part in causing more information processing to happen. 

You cite one paper (Chitka 2009) that provides ... (read more)

Curious what you're referring to here and if there's any publicly available information about it? Couldn't find anything in ALLFEDs 2020 and 2021 updates. (I'm trying to estimate the cost-effectiveness of this kind of project as part of my work at Rethink Priorities)

Another failure mode I couldn’t easily fit into the taxonomy that might warrant a new category:

Competency failures - EAs are just ineffective at achieving things in the world due to lack of skills (eg comms, politics, org running) or bad judgement. Maybe this could be classed as a resource failure (for failing to attract people with certain skills) or a rigor failure (for failing to develop them/learn from others). Will try to think of a title beginning with R…

Minor points:

• I was also considering something like value failures (EAs have the wrong moral the

Curious what people think of the argument that, given that people in the EA community have different rankings of the top causes, a close-to-optimal community outcome could be reached if individuals argmax using their own ranking?

(At least assuming that the number of people who rank a certain cause as the top one is proportional to how likely it is to be the top one.)

[Shortform version of this comment here.]

Update: I helped Linch collect data on the undergrad degrees of exceptionally successful people (using some of the ex post metrics Linch mentioned).

Of the 32 Turing Award winners in the last 20 years, 6 attended a top 10 US university, 16 attended another US university, 3 attended Oxbridge, and 7 attended other non-US universities. (full data)

Of the 97 Decacorn company founders I could find education data for, 19 attended a top 10 US university, 32 attended another US university, and 46 attended non-US universities ... (read more)

Tl;dr: Most Turing Award winners and Decacorn company founders (i.e., exceptionally successful people) don’t attend US top universities, but there’s a fair amount of concentration.

In response to the post Most Ivy-smart students aren't at Ivy-tier schools and as a follow-up to Linch’s comment tallying the educational background of Field Medalists, I collected some data on the undergrad degrees of exceptionally successful people (using some of the  (imperfect) ex post metrics suggested by Linch).

Of the 32 Turing Award winners in the last 20 years, ... (read more)

Thought-provoking post, thanks a lot for writing it! 

I broadly agree that it’s good for community builders to spend significant time on learning/direct work, especially if their long-term plan is not to do community building, but I think I disagree with some of your specific reasons.

I think the post sometimes conflates two senses of marketing. One is “pure” marketing, the other is marketing as you define it (i.e., marketing and ops), which includes things like organising content-heavy events and programs like fellowships. My instinct is that:

A. Most o... (read more)

No worries!

I don’t have strong opinions on a 4-week fellowship, no! I think my quick take would be that (a) it’s harder to teach the core EA ideas well in 4x1.5h sessions, (b) it’s harder to create a social community/have people become friends in 4 weeks, and (c) the group of people who’d commit to a 4-week program but not an 8-week program is relatively small, at least in a university group context. But I’m not too sure about this. It also seems plausible to me that 4 weeks could be better in  contexts like professional or city groups.

I’d be excited ... (read more)

Thanks for writing this post! I especially like the concrete alternatives with thoughtful upsides/downsides. As some others have said, I’d guess some of the downsides to the alternatives are quite significant, but would still love to see trials and to chat to anyone who runs trials.
 

A potentially useful alternative approach (especially for larger groups who can run multiple programs) is to have several alternative intro funnels at once. I.e. run the IF but also have a clear alternative for keen people with more background knowledge or who can quickly ... (read more)

Thanks for raising these points! A few of my (personal) reactions:

1. We definitely didn't intend for the post to presuppose that democracy is good for the long term. It’s true that most of the potential effects we identity are positive-leaning – but none of these effects, nor the all-things-considered effect, is a settled case. 

2. I think the question of what conditions allowed EA to come into existence is interesting, although not sure if that's the main positive impact of liberal democracy (especially given we don’t have super strong evidence that l... (read more)