(i.e. most people who are likely to update downwards on Yudkowsky on the basis of this post, seem to me to be generically too trusting, and I am confident I can write a more compelling post about any other central figure in Effective Altruism that would likely cause you to update downwards even more)
My impression is the post is somewhat unfortunate attempt to "patch" the situation in which many generically too trusting people updated a lot on AGI Ruin: A List of Lethalities and Death with Dignity and subsequent deference/update cascades.
I... (read more)
I. It might be worth reflecting upon how large part of this seem tied to something like "climbing the EA social ladder".
E.g. just from the first part, emphasis mine
... (read more)Coming to Berkeley and, e.g., running into someone impressive at an office space already establishes a certain level of trust since they know you aren’t some random person (you’ve come through all the filters from being a random EA to being at the office space).
If you’re in Berkeley for a while you can also build up more signals that you are worth people’s time. E.g., be involved in
Yeah, it would probably be good if people redirected this energy to climbing ladders in the government/civil service/military or important powerful corporate institutions. But I guess these ladders underpay you in terms of social credit/inner ringing within EA. Should we praise people aiming for 15y-to-high-impact careers more?
I agree with you, being "a highly cool and well networked EA" and "do things which need to be done" are different goals. This post is heavily influenced by my experience as a new community builder and my perception that, in this situation, being "a highly cool and well networked EA" and "do things which need to be done" are pretty similar. If I wasn't so sociable and network-y, I'd probably still be running my EA reading group with ~6 participants, which is nice but not "doing things which need to be done". For technical alignment researchers, this is probably less the case, though still much more than I would've expected.
(strongly upvoted because I think this is a clean explanation of what I think is an underrated point at the current stage, particularly among younger EAs).
Suggested variation, which I'd expect to lead to better results: use raw "completion probabilities" for different answers.
E.g. with prompt "Will Russia invade Ukrainian territory in 2022?" extract completion likelihoods of the next few tokes "Yes" and "No". Normalize
Also the direction of ALERT is generally more on "doing". Doing seems often very different from forecasting, often needs different people - part of the relevant skills is plausibly even anticorrelated.
Crisis response is a broader topic. I would probably suggest creating additional tag for Crises response (most of our recent sequence would fit there)
I don't have a strong preference. There a some aspects in which longerism can be better framing, at least sometimes.
I. In a "longetermist" framework, x-risk reduction is the most important thing to work on for many orders of magnitude of uncertainty about the probability of x-risk in the next e.g. 30 years. (due to the weight of the long term future). Even if AI related x-risk is only 10ˆ-3 in next 30 years, it is still an extremely important problem or the most important one. In a "short-termist" view with, say, a discount rate of 5%, it is not nearly so ... (read more)
Title EA Dropouts seems a bit confusing, because it can be naturally interpreted as people who dropped out of EA
I had little influence over the 1st wave, credit goes elsewhere.
What happened in subsequent waves is complicated. One sentence version is Czechia changed minister of health 4 times, only some of them were reasonably oriented, and how much they were interested in external advice differed a lot in time.
Note that the "death tolls per capita in the world" stats are misleading, due to differences in reporting. Czechia had average or even slightly lower than average mortality compared to "Eastern Europe" reference class, but much better reporting. For more reliable data, see https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(21)02796-3/fulltext
Both "EA translation service" and "EA spaces everywhere" seem like ideas which can take good, but also many bad or outright harmful forms.
A few years ago, I tried to describe how to to establish a robustly good "local effective altruism" in a new country or culture (other than Anglosphere).
The super brief summary is
1. it's not about translating a book, but about “transmission of a tradition of knowledge”
2. what is needed is a highly competent group of people
3. who can, apart from other things, figure out what the generative question of EA means... (read more)
Mass-reach posts came later, but sooner than the US mainstream updates
https://www.lesswrong.com/posts/h4vWsBBjASgiQ2pn6/credibility-of-the-cdc-on-sars-cov-2
https://slatestarcodex.com/2020/03/23/face-masks-much-more-than-you-wanted-to-know/
The practical tradeoff was between what, where and when to publish. The first version of the preprint which is on medrxive contains those estimates. Some version with them could probably be published in a much worse journal than Science, and would have much less impact.
We could have published them separately, but a paper is a lot of work, and it's not clear to me whether, for example, to sacrifice some of the"What we tried"and get this done would have been a good call.
It is possible to escape from the game in specific cases - in the case of covid, fo... (read more)
NPI & IFR: thanks, it's now explained in the text.
Re: Rigour
I think much of the problem is due not to our methods being "unrigourous" in any objective sense, but to interdisciplinarity. For example, in the survey case, we used mostly standard methods from a field called "discrete choice modelling" (btw, some EAs should learn it - it's a pretty significant body of knowledge on "how to determine people's utility functions").
Unfortunately, it's not something commonly found in the field of, for example, "mathematical modeling of infectious diseases"... (read more)
Hi, as the next post in the sequencd is about 'failures' I think it would be more useful after that is published.
Sorry, but this seems to me to confuse the topic of the post "Experimental Longtermism" and the topic of this post. Note that the posts are independent, and about different topics.
The table in this post is about timescales of OODA loops (observe–orient–decide–act), not about feedback. For example, in a situation which is unfolding on a timescale of days and weeks, as was the early response to covid, some actions are just too slow to have an impact: for example, writing a book, or funding basic research. The same is true for some decision and observat... (read more)
Note that CSER is running a project roughly in this direction.
Thanks for sharing! We plan to announce some new significant effort in Prague in next ~1 month, and also likely will offer some aid to people moving to Prague. If anyone is interested in moving somewhere in Europe, send me a pm.
Basic reasoning is Prague is pareto-optimal on some combination of 'existing community (already 10-20 FTE people working on longtermist projects, other ~10FTE job openings this year)', 'better quality of living', 'costs to relocate', 'in Europe', 'cultural sanity'. There wasn't much effort going into promoting Prague in ... (read more)
Millions is probably a safe bet/lower bound: majority won't be via direct twitter reads, but via mainstream media using it in their writing.
With twitter, we have a better overview in the case of our other research on seasonality (still in review!). Altmetric estimate is it was shared with accounts with an upper bound of 13M followers. However, in this case, almost all the shares were due to people retweeting my summary. Per twitter stats, it got 2M actual impressions. Given the fact the NPI research was shared and referenced more, it's probably ... (read more)
Handy reference! Apart from the average rate, it seems also interesting to notice the variance in the table, spread over 4 orders of magnitude. This may point to something like 'global sanity' being an important existential risk factor.
I mostly agree with the problem statement.
With the proposed solution of giving people feedback - I've historically proposed this on various occasions, and from what I have heard, one reason for not giving feedback on the side of organizations is something like "feedback opens up space for complaints, drama on social media, or even litigation". The problem looks very different from the side of the org: when evaluating hundreds of applications, it is basically certain some errors are made, some credentials misunderstood, experiences not counted as they shou... (read more)
My vague understanding is that there's likely no legal issues with giving feedback as long as it's impartial. It's instead one of those things where lawyers reasonably advise against doing anything not required since literally anything you do exposes you to risk. Of course you could give feedback that would obviously land you in trouble, e.g. "we didn't hire you because you're [ethnicity]/[gender]/[physical attribute]", but I think most people are smart enough to give feedback of the form "we didn't hire you because legible reason X".
And it's quickly becom... (read more)
I would guess the 'typical young researcher fallacy' also applies to Hinton - my impression is he is basically advising his past self, similarly to Toby. As a consequence, the advice is likely sensible for people-much-like-past-Hinton, but not a good general advice for everyone.
In ~3 years most people are able to re-train their intuitions a lot (which is part of the point!). This seems particularly dangerous in cases where expertise in the thing you are actually interested in does not exist, but expertise in something so... (read more)
Let's start with the third caveat: maybe the real crux is what we think are the best outputs; what I consider some of the best outputs by young researchers of AI alignment is easier to point at via examples - so it's e.g. the mesa-optimizers paper or multiple LW posts by John Wentworth. As far as I can tell, none of these seems to be following the proposed 'formula for successful early-career research'.
My impression is PhD students in AI in Berkeley need to optimise, and actually optimise a lot for success in an established field (ML/AI),... (read more)
It's good to see a new enthusiastic team working on this! My impression, based on working on the problem ~2 years ago is this has good chances to provide value in global health a poverty, animal suffering, or parts of meta- cause areas; in case of x-risk focused projects, something like a 'project platform' seems almost purely bottlenecked by vetting. In the current proposal this seems to mostly depend on "Evaluation Commission"-> as a result, the most important part for x-risk projects seems judgement of members of this commission and/or it's ability to seek external vetting
In my view this text should come with multiple caveats.
- Beware 'typical young researcher fallacy'. Young researchers are very diverse, and while some of them will benefit from the advice, some of them will not. I do not believe there is a general 'formula for successful early-career research'. Different people have different styles of doing research, and even different metrics for what 'successful research' means. While certainly many people would benefit from the advice 'your ideas are bad', some young researchers actually have great ideas, s... (read more)
I'm not going to go into much detail here, but I disagree with all of these caveats. I think this would be a worse post if it included the first and third caveats (less sure about the second).
First caveat: I think > 95% of incoming PhD students in AI at Berkeley have bad ideas (in the way this post uses the phrase). I predict that if you did a survey of people who have finished their PhD in AI at Berkeley, over 80% of them would think their initial ideas were significantly worse than their later ideas. (Note also that AI @ Berkeley is a very selective p... (read more)
Contrary to what seems an implicit premise of this post, my impression is
- most EA group organizers should have this as a side-project, and should not think about "community building" as about their "career path" where they could possibly continue to do it in a company like Salesforce
- the label "community building" is unfortunate for what most of the EA group organizing work should consist of
- most of the tasks in "EA community building" involve skills which are pretty universal a generally useable in most other fields, like "strategizin... (read more)
1.
For different take on very similar topic check this discussion between me and Ben Pace (my reasoning was based on the same Sinatra paper).
... (read more)
For practical purposes, in case of scientists, one of my conclusions was
Translating into the language of digging for gold, the prospectors differ in their speed and ability to extract gold from the deposits (Q). The gold in the deposits actually is randomly distributed. To extract exceptional value, you have to have both high Q and be very lucky. What is encouraging in selecting the talent is the Q se
First EuroSPARC was in 2016. Targeting 16-19 year olds, my prior would be participants should still mostly study, and not work full-time on EA, or only exceptionally.
Long feedback loops are certainly a disadvantage.
Also in the meantime ESPR underwent various changes and actually is not optimising for something like "conversion rate to an EA attractor state".
Quick reaction:
I. I did spent a considerable amount of time thinking about prioritisation (broadly understood)
My experience so far is
f... (read more)
I posted a short version of this, but I think people found it unhelpful, so I'm trying to post somewhat longer version.
I'm not sure you've read my posts on this topic? (1,2)
In the language used there, I don't think the groups you propose would help people overcome the minimum recommended resources, but are at the risk of creating the appearance some criteria vaguely in that direction are met.
FWIW the Why not to rush to translate effective altruism into other languages post was quite influential but is often wrong / misleading / advocating some very strong prior on inaction, in my opinion
I don't think this is actually neglected
(more on the topic here)
Sure
a)
For example, CAIS and something like "classical superintelligence in a box picture" disagree a lot on the surface level. However, if you look deeper, you will find many similar problems. Simple to explain example: problem of manipulating the operator - which has (in my view) some "hard core" involving both math and philosophy, where you want the AI to somehow communicate with humans in a way which at the same time allows a) the human to learn from the AI if the AI knows something about the world b) the operator's values are ... (read more)
I think the picture is somewhat correct, and we surprisingly should not be too concerned about the dynamic.
My model for this is:
1) there are some hard and somewhat nebulous problems "in the world"
2) people try to formalize them using various intuitions/framings/kinds of math; also using some "very deep priors"
3) the resulting agendas look at the surface level extremely different, and create the impression you have
but actually
4) if you understand multiple agendas deep enough, you get a sense
Thanks for the reply! Could you give examples of:
a) two agendas that seem to be "reflecting" the same underlying problem despite appearing very different superficially?
b) a "deep prior" that you think some agenda is (partially) based on, and how you would go about working out how deep it is?
Re: future of the program & ecosystem influences.
What bad things will happen if the program is just closed
As a side-note: In case of the Bay area, I'd expect some funding-displacement effects. BERI grant-making is strongly correlated with geography and historically BERI funded some things which could be classified as community building. LTFF is also somewhat Bay-centric, and also there seem to be some LTFF grants which could be hypothetically funded by several orgs. Also some things were likely funded informally by local philantrophists.
To make the model more realistic one should note
meta: I considered commenting, but instead I'm just flagging that I find it somewhat hard to have an open discussion about the EA hotel on the EA forum in the fundraising context. The feeling part is
Overall my impression is posting critical comments would be somewhat antisocial, posting just positives or endorsements is against good epistemics, so the personally safest thing to do for many is not to s... (read more)
Flagging that there has been a post specifically soliciting reasons against donating to the EA Hotel:
$100 Prize to Best Argument Against Donating to the EA Hotel
And also a Question which solicited critical responses:
Why is the EA Hotel having trouble fundraising?
I agree that the "equilibrium" you describe is not great, except I don't think it is an equilibrium; more that, due to various factors, things have been moving slower than they ideally should have.
EA hotel struggles to collect low tens of $
I'm guessing you meant tens-of-thousan... (read more)
I agree that the epistemic dynamics of discussions about the EA Hotel aren't optimal. I would guess that there are selection effects; that critics aren't heard to the same extent as supporters.
Relatedly, the amount of discussion about the EA Hotel relative to other projects may be a bit disproportionate. It's a relatively small project, but there are lots of posts about it (see OP). By contrast, there is far less discussion about larger EA orgs, large OpenPhil grants, etc. That seems a bit askew to my mind. One might wonder about the cost-effectiveness of relatively long discussions about small donations, given opportunity costs.
In practice, it's almost never the inly option - e.g. CZEA was able to find some private funding even before CBG existed; several other groups were at least partially professional before CBG. In general it's more like it's better if national-level groups are funded from EA
CZEA was able to find some private funding even before CBG existed
Interesting! Up until now, my intuition was that private funding is only feasible after the group has been around for a few years, gathered sufficient evidence for their impact and some (former student) members earn enough and donate to it (at least this was the case for EA Norway, as far as I know).
Somewhat off-topic, but if you have time, I'd be curious to hear how CZEA managed to secure early private funding. How long had CZEA been active when it first received funding, what kind ... (read more)
The reason may be somewhat simple: most AI alignment researchers do not participate (post or comment) on LW/AF or participate only a little. For more understanding why, check this post of Wei Dai and the discussion under it.
(Also: if you follow just LW, your understanding of the field of AI safety is likely somewhat distorted)
With hypothesis 4.&5. I expect at least Oli to have strong bias of being more enthusiastic in funding people who like to interact with LW (all other research qualities being equal), so I'm pretty sure it's not the case
2.... (read more)
The reason may be somewhat simple: most AI alignment researchers do not participate (post or comment) on LW/AF or participate only a little.
I'm wondering how many such people there are. Specifically, how many people (i) don't participate on LW/AF, (ii) don't already get paid for AI alignment work, and (iii) do seriously want to spend a significant amount of time working on AI alignment or already do so in their free time? (So I want to exclude researchers at organizations, random people who contact 80,000 Hours for advice on how to get involved, people
... (read more)In my experience teaching rationality is more tricky than the reference class education, and is an area which is kind of hard to communicate to non-specialists. One of the main reasons seems to be many people have somewhat illusory idea how much they understand the problem.
I've suggested something similar for happiness (https://www.lesswrong.com/posts/7Kv5cik4JWoayHYPD/nonlinear-perception-of-happiness ). If you don't want to introduce the weird asymmetry where negative counts and positive not, what you get out of that could be somewhat surprising - it possibly recovers more "common folk" altruism where helping people who are already quite well off could be good, and if you allow more speculative views on the space on mind-states, you are at risk of recovering something closely resembling some sort of "buddhist utilitarian calculus".
As humans, we are quite sensitive to signs of social approval and disapproval, and we have some 'elephant in the brain' motivation to seek social approval. This can sometimes mess up with epistemics.
The karma represents something like sentiment of people voting on a particular comment, weighted in a particular way. For me, this often did not seemed to be a signal adding any new information - when following the forum closely, usually I would have been able to predict what will get downvoted or upvoted.
What seemed problematic to me was 1. a numbe... (read more)
It's not an instance of complain, but take it as a datapoint: I've switched off the karma display on all comments and my experience improved. The karma system tends to mess up with my S1 processing.
It seems plausible karma is causing harm in some hard to perceive ways. (One specific way is by people updating on karma pattern mistaking them for some voice of the community / ea movement / ... )
I would expect if organizations working in the area have reviews of expected technologies and how they enable individuals to manufacture pathogens, which is likely the background necessary for constructing timelines, they would not publish too specific documents.
If people think this is generally good idea I would guess CZEA can make it running in few weeks. Most of the work likely comes from curating the content, not from setting up the service
To clarify - agree with the benefits of splitting the discussion threads for readability, but I was unenthusiastic about the motivation be voting.
I don't think karma/voting system should be given that much attention or should be used as a highly visible feedback on project funding.
I do think that it would help independently of that by allowing more focused discussion on individual issues.
If the main problem you want to solve is "scaling up grantmaking", there are probably many other ways how to do it other than "impact markets".
(Roughly, you can amplify any "expert panel of judges" evaluations with judgemental forecasting.)