In Twitter and elsewhere, I've seen a bunch of people argue that AI company execs and academics are only talking about AI existential risk because they want to manufacture concern to increase investments and/or as a distraction away from near-term risks and/or regulatory capture. This is obviously false.
However, there is a nearby argument that is likely true: which is that incentives drive how people talk about AI risk, as well as which specific regulations or interventions they ask for. This is likely to happen both explicitly and unconsciously. It's important (as always) to have extremely solid epistemics, and understand that even apparent allies may have (large) degrees of self-interest and motivated reasoning.
Safety-washing is a significant concern; similar things have happened a bunch in other fields, it likely has already happened a bunch in AI, and will likely happen again in the months and years to come, especially if/as policymakers and/or the general public become increasingly uneasy about AI.
Also I guess that current proposals would benefit openAI, google DeepMind and
Anthropic. If there becomes a need to register large training runs, they have
more money and infrastructure and smaller orgs would need to build that if they
wanted to compete. It just probably would benefit them.
As you say, I think that its wrong to say this is their primary aim (which other
CEOs would say there products might kill us all to achieve regulatory capture?)
but there is real benefit.
Contra claims like here and here, I think extraordinary evidence is rare for probabilities that are quasi-rational and mostly unbiased, and should be quite shocking when you see it. I'd be interested in writing an argument for why you should be somewhat surprised to see what I consider extraordinary evidence[1]. However, I don't think I understand the "for" case for extraordinary evidence being common[2], so I don't understand the case for it and can't present the best "against" case.
[1] operationalized e.g. as a 1000x or 10000x odds update on a question t... (read more)
Some categories where extraordinary evidence is common, off the top of my head:
* Sometimes someone knows the answer with high trustworthiness, e.g. I spend an
hour thinking about a math problem, fail to determine the answer, check the
answer, and massively update toward the textbook's answer and against others.
* Sometimes you have high credence that the truth will be something you
assigned very low credence to, e.g. what a stranger's full name is or who the
murderer is or what the winning lottery number is or what the next sentence
you hear will be.
* Maybe you meant to refer only to (binary) propositions (and exclude
unprivileged propositions like "the stranger's name is Mark Xu").
* Sometimes you update to 0 or 1 because of the nature of the proposition. E.g.
if the proposition is something like "(conditional on seeing Zach again) when
I next see Zach will he appear (to me) to be wearing a mostly-blue shirt."
When you see me it's impossible not to update infinitely strongly.
Separately, fwiw I endorse the Mark Xu post but agree with you that (there's a
very reasonable sense in which) extraordinary evidence is rare for stuff you
care about. Not sure you disagree with "extraordinary evidence is common"
proponents.
2
Linch
8d
Hmm I agree with those examples, the first one of which wasn't in my radar for
"sans a few broad categories of cases."
I especially agree with the "sometimes you can update to 0 or 1 because of the
nature of the proposition" for situations where you already have moderately high
probability in, and I find it uninteresting. This is possibly an issue with the
language of expressing things in odds ratios. So for the example of
maybe my prior probability was 20% -- 1:4 -- odds ratio, and my posterior
probability is like 99999:1. This ~400,000 factor update seems unproblematic to
me. So I want to exclude that.
I do want my operationalization to be more general than binary propositions. How
about this revised one:
Suppose I thought there was a 1/10,000 chance that the answer to a math question
is pi. And then I look at the back at the math textbook and the answer was pi.
I'd be like "huh that's odd." And if this happened several times in succession,
I could be reasonably confident that it's much more likely that my math
uncertainty is miscalibrated than that I just happen to get unlucky.
Similarly if I spent an hour considering the specific probability I get struck
by lightning tomorrow, or a specific sequence of numbers for the Powerball
tomorrow, and then was wrong, well that sure will be weird, and surprising.
2
Zach Stein-Perlman
8d
Minor note: I think it's kinda inelegant that your operationalization depends on
the kinds of question-answer pairs humans consider rather than asserting
something about the counterfactual where you consider an arbitrary
question-answer pair for an hour.
2
Linch
8d
Hmm I'm not sure I understand the inelegance remark, but I do want to
distinguish between something like --
which, while not technically excluded by the laws of probability, sure seems
wild if my beliefs are anything even approximately approaching a martingale --
from a situation like
I want to be careful to not borrow the credulity from the second case (a
situation that is natural, normal, commonplace under most formulations) and
apply to the first.
1. I think we should be more willing than we currently are to ban or softban people.
2. I think we should not assume that CEA's Community Health team "has everything covered"
3. I think more people should feel empowered to tell CEA CH about their concerns, even (especially?) if other people appear to not pay attention or do not think it's a major concern.
4. I think the community is responsible for helping the CEA CH team with having a stronger mandate to deal with interpersonal harm, including some degree of acceptance of mistakes of overzealous moderation.
(all views my own) I want to publicly register what I've said privately for a while:
For people (usually but not always men) who we have considerable suspicion that they've been responsible for significant direct harm within the community, we should be significantly more willing than we currently are to take on more actions and the associated tradeoffs of limiting their ability to cause more harm in the community.
Some of these actions may look pretty informal/unofficial (gossip, explicitly warning newcomers against specific people, keep an unofficial eye out for some people during par... (read more)
Thank you so much for laying out this view. I completely agree, including every single subpoint (except the ones about the male perspective which I don't have much of an opinion on). CEA has a pretty high bar for banning people. I'm in favour of lowering this bar as well as communicating more clearly that the bar is really high and therefore someone being part of the community certainly isn't evidence they are safe.
Thank you in particular for point D. I've never been quite sure how to express the same point and I haven't seen it written up elsewhere.
It's a bit unfortunate that we don't seem to have agreevote on shortforms.
As an aside, I dislike calling out gender like this, even with the "not always"
disclaimer. Compare: "For people (usually but not always black people)" would be
considered inappropriate.
2
Linch
6mo
Would you prefer "mostly but not always?"
I think the archetypal examples of things I'm calling out is sexual harassment
or abuse, so gender is unusually salient here.
0
MichaelDickens
6mo
I would prefer not to bring up gender at all. If someone commits sexual
harassment, it doesn't particularly matter what their gender is. And it may be
true that men do it more than women, but that's not really relevant, any more
than it would be relevant if black people committed sexual harassment more than
average.
8
James Özden
6mo
It's not that it "may be" true - it is true
[https://rapecrisis.org.uk/get-informed/statistics-sexual-violence/]. I think
it's totally relevant: if some class of people are consistently the perpetuators
of harm against another group, then surely we should be trying to figure out why
that it is the case so we can stop it? Not providing that information seems like
it could seriously impede our efforts to understand and address the problem (in
this case, sexism & patriarchy).
I'm also confused by your analogy to race - I think you're implying that it
would be discriminatory to mention race if talking about other bad things being
done, but I also feel like this is relevant. In this case I think it's a bit
different, however, as there's other confounders present (e.g. black people are
much more highly incarcerated, earn less on average, generally much less
privileged) which all might increase rates of doing said bad thing. So in this
case, it's not a result of their race, but rather a result of the unequal
socioeconomic conditions faced when someone is a certain race.
If your civilization's discount rate is too low, you'll mug yourself trying to prevent the heat death of the universe. If your discount rate is too high, you'll mug yourself wasting resources attempting time travel. The correct answer lies somewhere in between.
I think a subtext for some of the EA Forum discussions (particularly the more controversial/ideological ones) is that a) often two ideological camps form, b) many people in both camps are scared, c) ideology feeds on fear and d) people often don't realize they're afraid and cover it up in high-minded ideals (like "Justice" or "Truth").
I think if you think other EAs are obviously, clearly Wrong or Evil, it's probably helpful to
a) realize that your interlocutors (fellow EAs!) are human, and most of them are here because they want to serve the good
b) internally try to simulate their object-level arguments
c) try to understand the emotional anxieties that might have generated such arguments
d) internally check in on what fears you might have, as well as whether (from the outside, looking from 10,000 feet up) you might acting out the predictable moves of a particular Ideology.
e) take a deep breath and a step back, and think about your intentions for communicating.
I think this should be a full post. Happy to cowrite if you like.
2
ChanaMessinger
4mo
Strong +1 to everyone is scared (not literally, but I think it's true that many
people with a large range of opinions feel - potentially quite correctly - that
it's risky to speak up, and that can be missed if you're inhabiting one side of
that fear). I think I do better when thinking about c).
That said, while the writing I like second best takes c) seriously, I think the
writing I like best appears to almost forget that there are "sides" and just
kind of talks about the world.
I think I’m significantly more involved than most people I know in tying the fate of effective altruism in general, and Rethink Priorities in particular, with that of FTX. This probably led to rather bad consequences ex post, and I’m very sorry for this.
I don’t think I’m meaningfully responsible for the biggest potential issue with FTX. I was not aware of the alleged highly unethical behavior (including severe mismanagement of consumer funds) at FTX. I also have not, to my knowledge, contributed meaningfully to the relevant reputational laundering or branding that led innocent external depositors to put money in FTX. The lack of influence there is not because I had any relevant special knowledge of FTX, but because I have primarily focused on building an audience within the effective altruism community, who are typically skeptical of cryptocurrency, and because I have actively avoided encouraging others to invest in cryptocurrency. I’m personally pretty skeptical about the alleged social value of pure financialization in general and cryptocurrency in particular, and also I’ve always thought of crypto as a substantially more risky asset than many retail invest... (read more)
(The below does not necessarily excuse defects in awareness, understanding or
risk management around FTX/SBF from the most senior EA leaders, which should be
very sophisticated.)
Prior to November, the idea that FTX was strange or dangerous was not known to
even very experienced people in cryptocurrency.
As a datapoint from the non-EA world, several very smart people desperately
wanted to work for FTX, because of the status, culture and excitement around FTX
and SBF, even taking pay cuts to do so. For example, one such person was deeply
into crypto, seasoned and older, e.g. 800K TC L6+ at Google (higher in
salary/level to Jeff K for example).
This risk of clawback is probably truly unusual. I think this situation (massive
fraud leading to total collapse) is one of the only situations where this could
happen.
In replies to this thread, here are some thoughts I have around much of the discourse that have come out so far about recent controversies. By "discourse," I'm thinking of stuff I mostly see here, on EA Twitter, and EA Facebook. I will not opine on the details of the controversies themselves. Instead I have some thoughts on why I think the ensuing discourse is mostly quite bad, some attempts to reach an understanding, thoughts on how we can do better, as well as some hopefully relevant tangents.
I split my thoughts into multiple comments so people can upvote or downvote specific threads.
While I have thought about this question a bunch, these comments has been really hard for me to write and the final product is likely pretty far from what I’d like, so please bear with me. As usual, all errors are my own.
We (EA Forum) are maybe not strong enough (yet?) to talk about certain topics
A famous saying in LessWrong-speak is "Politics is the Mind-Killer". In context, the post was about attempting to avoid using political examples in non-political contexts, to avoid causing people to become less rational with political blinders on, and also to avoid making people with different politics feel unwelcome. More broadly, it's been taken by the community to mean a general injunction against talking about politics when unnecessary most of the time.
Likewise, I think there are topics that are as or substantially more triggering of ideological or tribal conflict as modern partisan politics. I do not think we are currently strong enough epistemically, or safe enough emotionally, to be able to discuss those topics with the appropriate level of nuance and intellect and tact. Except for the topics that are extremely decision-relevant (e.g. "which UK political party should I join to reduce malaria/factory farming/AI doom probabilities") I will personally prefer that we steer clear of them for now, and wait until our epistemics and cohesion are one day perhaps good enough to approach them.
I agree, but I think we may well never reach that point, in which case this is
tantamount to saying "never discuss it". And I think it's reasonable for people
who care about those issues to point out that we're ignoring them even though
they're important. Unsure how to resolve this.
3[anonymous]4mo
Honestly, a lot of the problems from politics stems from both it's totalizing
nature, comparable to strong longtermism, plus emotion hampers more often than
it helps in political discussions compared to longtermism.
I'd say that if EA can't handle politics in the general forum, then I think a
subforum for EA politics should be made. Discussions about the politics of EA or
how to effectively do politics can go there.
Meanwhile, the general EA forum can simply ban political posts and discussions.
Yes, it's a strong measure to ban politics here. But bluntly, in social
communities that want to have any level of civility and charity, ultimately it
does tend towards banning politics and discussion around it, except maybe in a
subforum.
5
Ben_West
4mo
Thanks for writing this, I think this is a helpful frame on some things that
have been annoying me about the Forum recently.
Some of my other comments have quite grandiose language and claims. In some ways this is warranted: the problems we face are quite hard. But in other ways perhaps the grandiosity is a stretch: we have had a recent surge of scandals, and we'll like have more scandals in the years and perhaps decades to come. We do need to be somewhat strong to face them well. But as Ozzie Gooen rightfully point out, in contrast to our historical moral heroes[1], the problems we face are pretty minor in comparison.
Nelson Mandela served 27 years in prison. Frederick Douglass was enslaved for twenty years. Abraham Lincoln faced some of the US's worst years, during which most of his children died, and just after he won the civil war, was assassinated.
In comparison, the problems of our movement just seems kind of small in comparison? "We kind of went down from two billionaires and very little political/social pushback, to one billionaire and very little political/social pushback?" A few people very close to us committed crimes? We had one of our intellectual heavyweights say something very racist 20+ years ago, and then apologized poorly? In the grand arc of ... (read more)
My big concern is that permanent harm could be suffered by either EA or it's
championed causes. Somewhat like how transhumanism became tarred with the brush
of racism and eugenics, I worry things like AI safety or X-risk work could be
viewed in the same light as racism. And there may be much more at stake than
people realize.
The problem is that even without a hinge of history, our impacts, especially in
a longtermism framework, are far far larger than previous generations, and we
could very well lose all that value if EA or it's causes become viewed as badly
as say eugenics or racism was.
I think many people on both sides of the discussion
have drawn bright lines way too quickly
were quick to assume bad faith from their interlocutors, understood people who disagreed with them as opponents or enemies in a culture war
rather than (possibly mistaken) collaborators in the pursuit of the good
mostly did not interpret their opponents particularly charitably
said things to denigrate their opponents rather than try to understand and empathize with them
appeared to have acted out of ideology or perhaps fear, rather than love
seem to mostly ignore the (frequently EA) bystanders to this ideological conflict, and ignore (or even delight in) the harm their words or other speech acts may have caused to bystanders.
Perhaps I’m just reasserting basic forum norms, but I think we should instead at least try to interpret other people on this forum more charitably. Moreover, I think we should generally try to be kind to our fellow EAs[1]. Most of us are here to do good. Many of us have made substantial sacrifices in order to do so. We may have some disagreements and misunderstandings now, and we likely will again in the future, but mora... (read more)
It’s easy to get jaded about this, but in many ways I find the EA community genuinely inspirational. I’m sometimes reminded of this when I hear about a new good thing EAs have done, and at EA Global, and when new EAs from far-away countries reach out to me with a specific research or career question. At heart, the EA community is a community of thousands of people, many of whom are here because they genuinely want to do the most good, impartially construed, and are actively willing to use reason and evidence to get there. This is important, and rare, and I think too easily forgotten.
I think it's helpful to think about a few things you're grateful for in the community (and perhaps even your specific interlocutors) before engaging in heated discourse.
In recent days, I've noticed an upsurge of talking at people rather than with them. I think there's something lost here, where people stopped assuming interlocutors are (possibly mistaken) fellow collaborators in the pursuit of doing good, but more like opponents to be shot down and minimized. I think something important is lost both socially and epistemically when we do this, and it's worthwhile to consider ways to adapt a more collaborative mindset. Some ideas:
1. Try to picture yourself in the other person's shoes. Try to understand, appreciate, and anticipate both their worries and their emotions before dashing off a comment.
2. Don't say "do better, please" to people you will not want to hear the same words from. It likely comes across as rather patronizing, and I doubt the background rates of people updating positively from statements like that is particularly high.
3. In general, start with the assumption of some basic symmetry on how and what types of feedback you'd like to receive before providing it to others.
I suspect at least some of the optics and epistemics around the recent controversies are somewhat manipulated by what I call "enemy action." That is, I think there are people out there who are not invested in this project of doing good, and are instead, for idiosyncratic reasons I don't fully understand[1], interested in taking our movement down. This distorts a) much of the optics around the recent controversies, b) much of the epistemics in what we talk about and what we choose to pay attention to and c) much of our internal sense of cohesion.
I don't have strong evidence of this, but I think it is plausible that at least some of the current voting on the forum on controversial issues is being manipulated by external actors in voting rings. I also think it is probable that some quotes from both on and off this forum are selectively mined in external sources, so if you come to the controversies from them, you should make take a step back and think of ways in which your epistemics or general sense of reality is being highjacked. Potential next steps:
Keep your cool
Assume good faith from community members most of the time
I've been thinking a lot about this, even before the FTX collapse but especially
since then. There are clearly some actors who are prioritising causing harm to
EA as one of their major goals. Separately, but related, is the fact that as EA
has grown the number of actors who view us negatively or as something to be
challenged/defeated has grown. This means that EA no longer acts in a neutral
information space.
Whatever we do from this point on there will be strong pushback. Some of it
might be justified, most of it (I hope) not, but regardless this is now
something I think we have to be more aware of. Bad actors can act against EA
unilaterally, and I have my doubts that the current approach
[https://forum.effectivealtruism.org/posts/onpvizbPgSrkigSjn/the-fidelity-model-of-spreading-ideas]
to spreading EA ideas may not be the best approach against this.
I may try to fill this out with more helpful ideas, concepts, and examples in a
top-level post.
I'm not sure this comment is decision-relevant, but I want us to consider the need for us, both individually and collectively, to become stronger. We face great problems ahead of us, and we may not be able up for the challenge. We need to face them with intellect, and care, and creativity and reason. We need to face them with cooperation, and cohesion, and love for fellow man, but also strong independence and skepticism and ability to call each out on BS.
We need to be clear enough in our thinking to identify the injustices in the world, careful enough in our planning to identify the best ways to fight them, and committed and steady enough in our actions to decisively act when we need to. We need to approach the world with fire in our hearts and ice in our veins.
We should try to help each other identify, grow, and embody the relevant abilities and virtues needed to solve the world's most pressing problems. We should try our best to help each other grow together.
This may not be enough, but we should at least try our best.
One basic lesson I learned from trying to do effective altruism for much of my adult life is that morality is hard. Morality is hard at all levels of abstraction: Cause prioritization, or trying to figure out the most pressing problems to work on, is hard. Intervention prioritization, or trying to figure out how we can tackle the most important problems to work on, is hard. Career choice, or trying to figure out what I personally should do to work on the most important interventions for the most important problems is hard. Day-to-day prioritization is hard. In practice, juggling a long and ill-defined list of desiderata to pick the morally least-bad outcome is hard. And dedication and commitment to continuously hammer away at doing the right thing is hard.
And the actual problems we face are really hard. Millions of children die every year from preventable causes. Hundreds of billions of animals are tortured in factory farms. Many of us believe that there are double-digit percentage points of existential risk this century. And if we can navigate all the perils and tribulations of this century, we still need to prepare our descendant... (read more)
Anthropic awareness or “you’re not just in traffic, you are traffic.”
An old standup comedy bit I like is "You're not in traffic, you are traffic."Traffic isn't just something that happens to you, but something you actively participate in (for example, by choosing to leave work during rush hour). Put another way, you are other people's traffic.
I take the generalized version of this point pretty seriously. Another example of this was I remember complaining about noise at a party. Soon after, I realized that the noise I was complaining about was just other people talking! And of course I participated in (and was complicit in) this issue.
Similarly, in recent months I complained to friends about the dropping kindness and epistemic standards on this forum. It took me way too long to realize the problem with that statement, but the reality is that discourse, like traffic, isn't something that just happens to me. If anything, as one of the most active users on this forum, I'm partially responsible for the dropping forum standards, especially if I don't active try to make epistemic standards better.
So this thread is my partial attempt to rectify the situation.
Understanding and acknowledging the subtext of fear
I think a subtext for some of the EA Forum discussions (particularly the more controversial/ideological ones) is that a) often two ideological camps form, b) many people in both camps are scared, c) ideology feeds on fear and d) people often don't realize they're afraid and cover it up in high-minded ideals (like "Justice" or "Truth")[1].
I think if you think other EAs are obviously, clearly Wrong or Evil, it's probably helpful to
a) realize that your interlocutors (fellow EAs!) are human, and most of them are here because they want to serve the good
b) internally try to simulate their object-level arguments
c) try to understand the emotional anxieties that might have generated such arguments
d) internally check in on what fears you might have, as well as whether (from the outside, looking from 10,000 feet up) you might acting out the predictable moves of a particular Ideology.
e) take a deep breath and a step back, and think about your intentions for communicating.
When embroiled in ideological conflict, I think it's far too easy to be ignorant of (or in some cases, deliberately downplay for bravado reasons) the existence of bystanders to your ideological war. For example, I think some black EAs are directly hurt by the lack of social sensitivity displayed in much of the discourse around the Bostrom controversy (and perhaps the discussions themselves). Similarly, some neurodivergent people are hurt by the implication that maximally sensitive language is a desiderata on the forum, and the related implication that people like them are not welcome. Controversies can also create headaches for community builders (including far away from the original controversy), for employees at the affected or affiliated organizations, and for communications people more broadly.
The move to be making is to stop for a bit. Note that people hurting are real people, not props. And real people could be seriously hurting for reasons other than direct ideological disagreement.
While I think it is tempting to use bystanders to make your rhetoric stronger, embroiling bystanders in your conflict is I think predictably bad. If you know people who you think m... (read more)
The whole/only real point of the effective altruism community is to do the most good.
If the continued existence of the community does the most good, I desire to believe that the continued existence of the community does the most good; If ending the community does the most good, I desire to believe that ending the community does the most good; Let me not become attached to beliefs I may not want.
I think a plausibly good training exercise for EAs wanting to be better at empirical/conceptual research is to deep dive into seminal papers/blog posts and attempt to identify all the empirical and conceptual errors in past work, especially writings by either a) other respected EAs or b) other stuff that we otherwise think of as especially important.
I'm not sure how knowledgeable you have to be to do this well, but I suspect it's approachable for smart people who finish high school, and certainly by the time they finish undergrad^ with a decent science or social science degree.
I think this is good career building for various reasons:
you can develop a healthy skepticism of the existing EA orthodoxy
I mean skepticism that's grounded in specific beliefs about why things ought to be different, rather than just vague "weirdness heuristics" or feeling like the goals of EA conflict with other tribal goals.
(I personally have not found high-level critiques of EA, and I have read many, to be particularly interesting or insightful, but this is just a personal take).
you actually deeply understand at least one topic well enough
This is another example of a Shortform that could be an excellent top-level post (especially as it's on-theme with the motivated reasoning post that was just published). I'd love to see see this spend a week on the front page and perhaps convince some readers to try doing some red-teaming for themselves. Would you consider creating a post?
Strong upvote for a idea that seems directly actionable and useful for
addressing important problem.
I'm gonna quote your shortform in full (with a link and attribution, obviously)
in a comment on my post about Intervention options for improving the EA-aligned
research pipeline
[https://forum.effectivealtruism.org/posts/MxXQ2bbL6KPrHDPtz/intervention-options-for-improving-the-ea-aligned-research].
I think by default good ideas like this never really end up happening, which is
sad. Do you or other people have thoughts on how to make your idea actually
happen? Some quick thoughts from me:
* Just highlight this idea on the Forum more often/prominently
* People giving career advice or mentorship to people interested in EA-aligned
research careers mention this as one way of testing fit, having an impact,
etc.
* I add the idea to Notes on EA-related research, writing, testing fit,
learning, and the Forum
[https://forum.effectivealtruism.org/posts/J7PsetipHFoj2Mv7R/notes-on-ea-related-research-writing-testing-fit-learning]
[done!]
* Heap an appropriate amount of status and attention on good instances of this
having been done
* That requires it to be done at least once first, of course, but can then
increase the rate
* E.g., Aaron Gertler could feature it in the EA Forum Digest newsletter,
people could make positive comments on the post, someone can share it in a
Facebook group and/or other newsletter
* I know I found this sort of thing a useful and motivating signal when I
started posting stuff (though not precisely this kind of stuff)
* Publicly offer to provide financial prizes for good instances of this having
been done
* One way to do this could mirror Buck's idea for getting more good book
reviews to happen (see my other comment): "If it’s the kind of review I
want, I give them $500 in return for them posting the review to EA Forum or
LW with a “This post sponsored by the EAIF” banner at t
2
MichaelA
2y
Another possible (but less realistic?) way to make this happen:
* Organisations/researchers do something like encouraging red teaming of their
own output, setting up a bounty/prize for high-quality instances of that, or
similar
* An example of something roughly like this is a post on the GiveWell blog
[https://blog.givewell.org/2013/01/23/guest-post-from-david-barry-about-deworming-cost-effectiveness/]
that says at the start: "This is a guest post by David Barry
[http://pappubahry.livejournal.com/], a GiveWell supporter. He emailed us
at the end of December to point out some mistakes and issues in our
cost-effectiveness calculations for deworming, and we asked him to write up
his thoughts to share here. We made minor wording and organizational
suggestions but have otherwise published as is; we have not vetted his
sources or his modifications to our spreadsheet for comparing deworming and
cash. Note that since receiving his initial email, we have discussed the
possibility of paying him to do more work like this in the future."
* But I think GiveWell haven't done that since then?
* It seems like this might make sense and be mutually beneficial
* Orgs/researchers presumably want more ways to increase the accuracy of
their claims and conclusions
* A good red teaming of their work might also highlight additional
directions for further research and surface someone who'd be a good
employee for that org or collaborator for that researcher
* Red teaming of that work might provide a way for people to build skills
and test fit for work on precisely the topics that the org/researcher
presumably considers important and wants more people working on
* But I'd guess that this is unlikely to happen in this form
* I think this is mainly due to inertia plus people feeling averse to the
idea
* But there may also be good arguments against
* Th
4
Linch
2y
Yeah I think this is key. I'm much more optimistic about getting trainees to do
this being a good training intervention than a "directly improve research
quality" intervention. There are some related arguments why you want to pay
people who are either a) already good at the relevant work or b) specialized
reviewers/red-teamers
1. paying people to criticize your work would risk creating a weird power
dynamic, and more experienced reviewers would be better at navigating this
1. For example, trainees may be afraid of criticizing you too harshly.
2. Also, if the critique is in fact bad, you may be placed in a somewhat
awkward position when deciding whether to publish/publicize it.
4
Linch
2y
As you know, one of my interns is doing something adjacent to this idea (though
framed in a different way), and I may convince another intern to do something
similar (depending on their interests and specific project ideas in mind).
2
MichaelA
2y
Yeah, good point - I guess a more directed version of "People giving career
advice or mentorship to people interested in EA-aligned research careers mention
this as one way of testing fit, having impact, etc." is just people encouraging
people they manage to do this, or maybe even hiring people with this partly in
mind.
Though I think that that wouldn't capture most of the potential value of this
idea, since part of what's good about is that, as you say, this idea:
(People who've already gone through a hiring process and have an at least
somewhat more experienced researcher managing them will have an easier time than
other people in testing fit, having impact, building skills, etc. in other ways
as well.)
2
Linch
2y
Yeah I agree that a major upside to this idea (and a key differentiator between
it and other proposed interventions for fixing early stages of the research
pipeline) is that it ought to be doable without as much guidance from external
mentors. I guess my own willingness to suggest this as an intern project
suggests that I believe it must comparatively be even more exciting for people
without external guidance.
4
Linch
2y
Thanks for linking my idea in your sequence! (onlookers note: MichaelA and I are
coworkers)
This arguably happened
[https://forum.effectivealtruism.org/posts/zrgHTwBxyFFHfKaCK/ea-forum-prize-winners-for-january-2021]
to alexrjl's critique of Giving Green
[https://forum.effectivealtruism.org/posts/7yN7SKPpL3zN7yfcM/why-i-m-concerned-about-giving-green],
though it was a conjunction of a critique of an organization and a critique of
research done.
As an aside, I decided to focus my shortform on critiques of public research
rather than critiques of organizations/people, even though I think the latter is
quite valuable too, since a) my intuition is that the former is less
acrimonious, b) relatedly, critiques of organizations may be worse at training
dispassionate analysis skills (vs eg tribalistic feelings or rhetoric), c)
critiques of orgs or people might be easier for newbies to fuck up and d) I
think empirically, critiques of organizations have a worse hit rate than
critiques of research posts.
5
Max_Daniel
2y
I would be very excited about someone experimenting with this and writing up the
results. (And would be happy to provide EAIF funding for this if I thought the
details of the experiment were good and the person a good fit for doing this.)
If I had had more time, I would have done this for the EA In-Depth Fellowship
seminars I designed and piloted recently.
I would be particularly interested in doing this for cases where there is some
amount of easily transmissible 'ground truth' people can use as feedback signal.
E.g.
* You first let people red-team deworming papers and then give them some more
nuanced 'Worm Wars' stuff
[https://blog.givewell.org/2016/12/06/why-i-mostly-believe-in-worms/]. (Where
ideally you want people to figure out "okay, despite paper X making that
claim we shouldn't believe that deworming helps with short/mid-term education
outcomes, but despite all the skepticism by epidemiologists here is why it's
still a great philanthropic bet overall" - or whatever we think the
appropriate conclusion is.)
* You first let people red-team particular claims about the effects on hen
welfare from battery cages vs. cage-free environments and then you show them
Ajeya's report
[https://www.openphilanthropy.org/focus/us-policy/farm-animal-welfare/how-will-hen-welfare-be-impacted-transition-cage-free-housing].
* You first let people red-team particular claims about the impacts of the
Justinian plague and then you show them this paper
[https://www.pnas.org/content/116/51/25546].
* You first let people red-team particular claims about "X is power-law
distributed" and then you show them Clauset et al., Power-law distributions
in empirical data [https://arxiv.org/abs/0706.1062].
(Collecting a list of such examples would be another thing I'd be potentially
interested to fund.)
4
Linch
2y
Hmm I feel more uneasy about the truthiness grounds of considering some of these
examples as "ground truth" (except maybe the Clauset et al example, not sure).
I'd rather either a) train people to Red Team existing EA orthodoxy stuff and
let their own internal senses + mentor guidance decide whether the red teaming
is credible or b) for basic scientific literacy stuff where you do want clear
ground truths, let them challenge stuff that's closer to obvious junk (Why We
Sleep, some climate science stuff, maybe some covid papers, maybe pull up
examples from Calling Bullshit
[https://www.amazon.com/Calling-Bullshit-Skepticism-Data-Driven-World/dp/0525509186],
which I have not read).
4
Max_Daniel
2y
That seems fair. To be clear, I think "ground truth" isn't the exact framing I'd
want to use, and overall I think the best version of such an exercise would
encourage some degree of skepticism about the alleged 'better' answer as well.
Assuming it's framed well, I think there are both upsides and downsides to using
examples that are closer to EA vs. clearer-cut. I'm uncertain on what seemed
better overall if I could only do one of them.
Another advantage of my suggestion in my view is that it relies less on mentors.
I'm concerned that having mentors that are less epistemically savvy than the
best participants can detract a lot from the optimal value that exercise might
provide, and that it would be super hard to ensure adequate mentor quality for
some audiences I'd want to use this exercise for. Even if you're less concerned
about this, relying on any kind of plausible mentor seems like less scaleable
than a version that only relies on access to published material.
Upon (brief) reflection I agree that relying on the epistemic savviness of the mentors might be too much and the best version of the training program will train a sort of keen internal sense of scientific skepticism that's not particularly reliant on social approval.
If we have enough time I would float a version of a course that slowly goes from very obvious crap (marketing tripe, bad graphs) into things that are subtler crap (Why We Sleep, Bem ESP stuff) into weasely/motivated stuff (Hickel? Pinker? Sunstein? popular nonfiction in general?) into things that are genuinely hard judgment calls (papers/blog posts/claims accepted by current elite EA consensus).
But maybe I'm just remaking the Calling Bullshit course but with a higher endpoint.
___
(I also think it's plausible/likely that my original program of just giving somebody an EA-approved paper + say 2 weeks to try their best to Red Team it will produce interesting results, even without all these training wheels).
This idea sounds really cool. Brainstorming: a variant could be several people
red teaming the same paper and not conferring until the end.
5
MichaelA
2y
This also reminds me of a recent shortform by Buck
[https://forum.effectivealtruism.org/posts/Soutcw6ccs8xxyD7v/buck-s-shortform?commentId=LoSrFqCRuAyypeNvm]:
(I think the full shortform and the comments below it are also worth reading.)
One additional risk: if done poorly, harsh criticism of someone else's blog post from several years ago could be pretty unpleasant and make the EA community seem less friendly.
I'm actually super excited about this idea though - let's set some courtesy norms around contacting the author privately before red-teaming their paper and then get going!
I think I agree this is a concern. But just so we're on the same page here,
what's your threat model? Are you more worried about
1. The EA community feeling less pleasant and friendly to existing established
EAs, so we'll have more retention issues with people disengaging?
2. The EA community feeling less pleasant and friendly to newcomers, so we have
more issues with recruitment and people getting excited to join novel
projects?
3. Criticism makes being open about your work less pleasant, and open Red
Teaming about EA projects makes EA move even further in the direction of
being less open than we used to be. See also Responsible Transparency
Consumption
[https://www.jefftk.com/p/responsible-transparency-consumption].
4. Something else?
5
Kirsten
2y
It's actually a bit of numbers 1-3; I'm imagining decreased engagement
generally, especially sharing ideas transparently.
6
Linch
2y
Thanks for the excitement! I agree that contacting someone ahead of time might
be good (so at least someone doesn't learn about their project being red teamed
until social media blows up), but I feel like it might not mitigate most of the
potential unpleasantness/harshness. Like I don't see a good cultural way to both
incentivize Red Teaming and allow a face-saving way to refuse to let your papers
be Red Teamed.
Like if Red Teaming is opt-in by default, I'd worry a lot about this not taking
off the ground, while if Red Teaming is opt-out by default I'd just find it very
suss for anybody to refuse (speaking for myself, I can't imagine ever refusing
Red Teaming even if I would rather it not happen).
7
JJ Hepburn
2y
1. Easy steps could be to add a "red team" tag on the forum and point to this
post to encourage people to do this.
2. I have at times given advice to early career EA's mostly in AI Safety
similar to this. When people have trouble coming up with something they
might want to write about on the forum, I encourage them to look for the
things they don't think are true. Most people are passively reading the
forum anyway but actively looking for something the reader doesn't think is
true or is unconvinced by can be a good starting point for a post. It may be
that they end up convinced of the point but can still write a post making is
clearer and adding the arguments they found.
1. Having said this, most peoples first reaction is a terrified look.
Encouraging someone's first post to be a criticism is understandably
scary.
3. It may be hard to get both the benefit to the participants and to the orgs.
Anyone not intimidated by this might already have enough experience and
career capital. To give juniors the experience you might have to make it
more comfortable school work where the paper is written but only read by one
other person. This makes it harder to capture the career capital.
4. I'd expect this to be unlikely for someone to do individually and of their
own accord. At the very least best to do this in small groups to create
social accountability and commitment pressures. While also defusing the
intimidation. Alternately part of an existing program like an EA Fellowship.
Even better as it's own program, with all the overhead that comes with that.
4
MichaelA
2y
I think your cons are good things to have noted, but here are reasons why two of
them might matter less than one might think:
* I think the very fact that "It's possible that doing deliberate "red-teaming"
would make one predisposed to spot trivial issues rather than serious ones,
or falsely identify issues where there aren't any" could actually also make
this useful for skill-building and testing fit; people will be forced to
learn to avoid those failure modes, and "we" (the community, potential future
hirers, etc.) can see how well they do so.
* E.g., to do this red teaming well, they may have to learn to identify how
central an error is to a paper/post's argument, to think about whether a
slightly different argument could reach the same conclusion without needing
the questionable premise, etc.
* I have personally found that the line between "noticing errors in existing
work" and "generating novel research" is pretty blurry.
* A decent amount of the research I've done (especially some that is
unfortunately nonpublic so far) has basically followed the following
steps:
1. "This paper/post/argument seems interesting and important"
2. "Oh wait, it actually requires a premise that they haven't noted and
that seems questionable" / "It ignores some other pathway by which a bad
thing can happen" / "Its concepts/definitions are fuzzy or conflate
things in way that may obscure something important"
3. [I write a post/doc that discusses that issue, provides some analysis in
light of this additional premise being required or this other pathway
being possible or whatever, and discussing what implications this has -
e.g., whether some risk is actually more or less important than we
thought, or what new intervention ideas this alternative risk pathway
suggests might be useful]
* Off the top of my head, some useful pieces of public work by other p
General suspicion of the move away from expected-value calculations and cost-effectiveness analyses.
This is a portion taken from a (forthcoming) post about some potential biases and mistakes in effective altruism that I've analyzed via looking at cost-effectiveness analysis. Here, I argue that the general move (at least outside of human and animal neartermism) away from Fermi estimates, expected values, and other calculations just makes those biases harder to see, rather than fix the original biases.
I may delete this section from the actual post as this point might be a distraction from the overall point.
____
I’m sure there are very good reasons (some stated, some unstated) for moving away from cost-effectiveness analysis. But I’m overall pretty suspicious of the general move, for a similar reason that I’d be suspicious of non-EAs telling me that we shouldn’t use cost-effectiveness analyses to judge their work, in favor of say systematic approaches, good intuitions, and specific contexts like lived experiences (cf. Beware Isolated Demands for Rigor):
I’m sure you have specific arguments for why in your case quantitative approaches aren’t very necessary and useful, because your uncert
What kinds of things do you think it would be helpful to do cost effectiveness
analyses of? Are you looking for cost effectiveness analyses of problem areas or
specific interventions?
I think it would be valuable to see quantitative estimates of more problem areas and interventions. My order of magnitude estimate would be that if one is considering spending $10,000-$100,000, one should do a simple scale, neglectedness, and tractability analysis. But if one is considering spending $100,000-$1 million, one should do an actual cost-effectiveness analysis. So candidates here would be wild animal welfare, approval voting, improving institutional decision-making, climate change from an existential risk perspective, biodiversity from an existential risk perspective, governance of outer space etc. Though it is a significant amount of work to get a cost-effectiveness analysis up to peer review publishable quality (which we have found requires moving beyond Guesstimate, e.g. here and here), I still think that there is value in doing a rougher Guesstimate model and having a discussion about parameters. One could even add to one of our Guesstimate models, allowing a direct comparison with AGI safety and resilient foods or interventions for loss of electricity/industry from a long-term perspective.
I agree with the general flavor of what you said, but am unsure about the exact
numbers.
7
Linch
2y
Hmm one recent example is that somebody casually floated to me an idea that can
potentially entirely solve an existential risk (though the solution might have
downside risks of its own) and I realized then that I had no idea how much to
price the solution in terms of EA $s, like whether it should be closer to 100M,
1B or $100B.
My first gut instinct was to examine the solution and also to probe the downside
risks, but then I realized this is thinking about it entirely backwards. The
downside risks and operational details don't matter if even the most optimistic
cost-effectiveness analyses isn't enough to warrant this being worth funding!
An org or team of people dedicated to Red Teaming EA research. Can include checks for both factual errors and conceptual ones. Like JEPSEN but for research from/within EA orgs. Maybe start with one trusted person and then expand outwards.
After demonstrating impact/accuracy for say 6 months, can become a "security" consultancy for either a) EA orgs interested in testing the validity of their own research or b) an external impact consultancy for the EA community/EA donors interested in testing or even doing the impact assessments of specific EA orgs. For a), I imagine Rethink Priorities may want to become a customer (speaking for myself, not the org).
Potentially good starting places:
- Carefully comb every chapter of The Precipice
- Go through ML/AI Safety papers and (after filtering on something like prestige or citation count) pick some papers at random to Red Team
- All of Tetlock's research on forecasting, particularly the ones with factoids most frequently cited in EA circle... (read more)
Honestly I don't understand the mentality of being skeptical of lots of spending on EA outreach. Didn't we have the fight about overhead ratios, fundraising costs, etc with Charity Navigator many years ago? (and afaict decisively won).
If I had to steelman it, perhaps groups constantly have to fight against the
natural entropy of spending more and more on outreach and less and less on
object-level work. And perhaps it is a train that once you start is hard to
stop: if the movement builders are optimising for growth, then the new people
they train will optimise for it as well and then there becomes an entrenched
class of movement-builders whose livelihood depends on the spending continuing
to remain and whose career prospects will be better if it further grows.
2
Kirsten
8mo
I recall the conclusion being, "Overhead costs aren't a good way to measure the
effectiveness of a charity," rather than anything stronger.
2
Linch
8mo
To me, I think the main thing is to judge effectiveness by outcomes, rather than
by processes or inputs.
2
rodeo_flagellum
8mo
While I have much less experience in this domain, i.e. EA outreach, than you, I
too fall on the side of debate that the amount spent is justified, or at least
not negative in value. Even if those who've learned about EA or who've
contributed to it in some way don't identify with EA completely, it seems that
in the majority of instances some benefit was had collectively, be it from the
skepticism, feedback, and input of these people on the EA movement / doing good
or from the learning and resources the person tapped into and benefited from by
being exposed to EA.
5
MichaelStJules
8mo
Is the concern more with how it's spent? Paying people fairly to work on
outreach seems good. Paying for food at some outreach events just to bring
people in for the first time seems good. Spending much more this way seems good,
as long as it actually contributes to the overall good done.
However, if we're spending on things that seem "luxurious", this could encourage
careless spending, corruption and putting your own pleasure, status or material
wealth over doing good for others, and could attract grifters and selfish
individuals to EA. I'm not sure exactly where that bar would be met. Maybe
paying for food for regulars at every EA meet-up, especially more expensive
food. Maybe paying for people who aren't yet working on any EA-related things to
attend conferences like EAG without specific expectations of work in return
(even local organizing or volunteering at EAG) might also carry such risks, but
1. I'm not sure to what extent that's really happening.
2. Those risks would exist even if we expected work in return or only let in
people already working on EA things. People may volunteer abroad for the
experience and to check off boxes, instead of actually doing good, so the
same could happen in EA.
3. Our screening for EA potential for EAGs might be good enough, anyway.
4. Maybe it's fine if people are a little selfish, as long as they would
contribute overall.
1
sphor
8mo
Most of the criticisms I've seen about EA spending on outreach seem driven not
by the conceptual reasons you mention, but empirical estimates of how effective
EA spending on outreach actually is.
2
Linch
8mo
Could you give examples? Usually the arguments I see look more like " Does it
really make sense to pay recent college grads $X" or "isn't flying out college
students to international conferences kinda extravagant?"and not "the EV of this
grant is too low relative to the costs."
1
sphor
8mo
I don't have any particular examples in mind, just speaking about my gestalt
impression. My impression is that criticisms like "Does it really make sense to
pay recent grads $X" and "flying college students out to international
conferences is extravagant" are usually fundamentally based on
(implicit/unconscious, not necessarily in-depth) estimates that these activities
are not in fact high EV relative to costs/alternatives, just phrased in
different ways and stressing different parts than EAs typically do. I believe
many people making these criticisms would not make them (or would moderate them
significantly) if they thought that the activities were high EV even if the
overhead ratio or fundraising costs were high.
One thing I dislike about certain thought-experiments (and empirical experiments!) is that they do not cleanly differentiate between actions that are best done in "player vs player" and "player vs environment" domains.
For example, a lot of the force of our intuitions behind Pascal's mugging comes from wanting to avoid being "mugged" (ie, predictably lose resources to an adversarial and probably-lying entity). However, most people frame it as a question about small probabilities and large payoffs, without the adversarial component.
Similarly, empirical social psych experiments on hyperbolic discounting feel suspicious to me. Indifference between receiving $15 immediately vs $30 in a week (but less aggressive differences between 30 weeks and 31 weeks) might track a real difference in discount rates across time, or it could be people's System 1 being naturally suspicious that the experimenters would actually pay up a week from now (as opposed to immediately).
So generally I think people should be careful in thinking about, and potentially cleanly differentiating, the "best policy for making decisions in normal contexts" vs "best policy for making decisions in contexts where someone is actively out to get you."
One thing that confuses me is that the people saying "EAs should be more frugal" and the people saying "EAs should be more hardworking" are usually not the same people. This is surprising to me, since I would have guessed that answers to considerations like "how much should we care about the demands of morality" and "how much should we trade off first order impact for inclusivity" and "how much we should police fellow travelers instead of have more of a live-and-let-live attitude about the community" should cluster pretty closely together.
lol why was this downvoted?
EDIT: Unfortunately shortforms don't have agree/disagree votes, so I can't tell
if people downvoted because they disagreed with the conclusion or because they
thought the observation is poorly argued.
A general policy I've adapted recently as I've gotten more explicit* power/authority than I'm used to is to generally "operate with slightly to moderately more integrity than I project explicit reasoning or cost-benefits analysis would suggest."
This is primarily for epistemics and community epistemics reasons, but secondarily for optics reasons.
I think this almost certainly does risk leaving value on the table, but on balance it is a better balance than potential alternatives:
Just following explicit reasoning likely leads to systematic biases "shading" the upsides higher and the downsides lower, and I think this is an explicit epistemics bias that can and should be corrected for.
there is also a slightly adversarial dynamics on the optics framing -- moves that seem like a normal/correct amount of integrity to me may adversarially be read as lower integrity to others.
Projections/forecasts of reasoning (which is necessary because explicit reasoning is often too slow) may additionally be biased on top of the explicit reasoning (I have some pointers here)
Always "behaving with maximal integrity" probably leaves too much value on the table, unless you define integrity in a pre
Target audience: urgent longtermists, particularly junior researchers and others who a) would benefit from more conceptual clarity on core LT issues, and b) haven’t thought about them very deeply.
Note that this shortform assumes but does not make arguments about a) the case for longtermism or b) the case for urgent (vs patient) longtermism, or c) the case that the probability of avertable existential risk this century is fairly high. It probably assumes other assumptions that are commonly held in EA as well.
___
Thinking about protecting the future in terms of extinctions, dooms, and utopias
When I talk about plans to avert existential risk with junior longtermist researchers and others, I notice many people, myself included, being somewhat confused about what we actually mean when we talk in terms of “averting existential risk” or “protecting the future.” I notice 3 different clusters of definitions that people have intuitive slippage between, where it might help to be more concrete:
1. Extinction – all humans and our moral descendants dying
2. Doom - drastic and irrevocable curtailing of our potential (This is approximately the standard definition)
The pedant in me wants to ask to point out that your third definition doesn’t
seem to be a definition of existential risk? You say —
It does make (grammatical) sense to define existential risk as the "drastic and
irrevocable curtailing of our potential". But I don’t think it makes sense to
literally define existential risk as “(Not) on track to getting to the best
possible future, or only within a small fraction of value away from the best
possible future.”
A couple definitions that might make sense, building on what you wrote:
* A sudden or drastic reduction in P(Utopia)
* A sudden or drastic reduction in the expected value of the future
* The chance that we will not reach ≈ the best futures open to us
I feel like I want to say that it's maybe a desirable featured of the term
'existential risk' that it's not so general to encompass things like "the
overall risk that we don't reach utopia", such that slowly steering towards the
best futures would count as reducing existential risk. In part this is because
most people's understanding of "risk" and certainly of "catastrophe" involve
something discrete and relatively sudden.
I'm fine with some efforts to improve P(utopia) not being counted as efforts to
reduce existential risk, or equivalently the chance of existential catastrophe.
And I'd be interested in new terminology if you think there's some space of
interventions that aren't neatly captured by that standard definitions of
existential risk.
2
Linch
1y
Yeah I think you raise a good point. After I wrote the shortform (and after our
initial discussion), I now lean more towards just defining "existential risk" as
something in the cluster of "reducing P(doom)" and treat alternative methods of
increasing the probability of utopia as a separate consideration.
I still think highlighting the difference is valuable. For example, I know
others disagree, and consider (e.g) theoretically non-irrevocable flawed
realizations as form of existential risk even in the classical sense.
3
Ivy Mazzola
1y
Just scanning shortform for kicks and see this. Good thoughts and I find myself
cringeing at the term "existential risk" often because of what you say about
extinction, and wishing people spoke about utopia.
Can I ask your reasoning for putting this in shortform? I've seen pieces on the
main forum with much less substance. I hope you write something else up about
this. I think utopia framework could also be good for community mental health,
while for many people still prompting them to the same career path and other
engagement.
2
Linch
1y
Thanks for your kind words!
I don't have a strong opinion about this, but I think of shortforms as more
quickly dashed off thoughts, while frontpage posts have (relatively) more polish
or novelty or both.
Another thing is that this shortform feels like a "grand vision" thing, and I
feel like main blog posts that talk about grand visions/what the entire
community should do demand more "justification" than either a) shortforms that
talk about grand visions/what the entire community should do or b) main blog
posts that are more tightly scoped. And I didn't provide enough of such
justification.
Another consideration that jumps to mind is something about things in the
shortform being similar to my job but not quite my job, and me not wanting to
mislead potential employees, collaborators, etc, about what working on
longtermism at Rethink is like. (This is especially an issue because I don't
post job outputs here that often, so a marginal post is more likely to be
misleading). Not sure how much to weigh this consideration, but it did pop to
mind.
I'm pretty confused about the question of standards in EA. Specifically, how high should it be? How do we trade off extremely high evidential standards against quantity, either by asking people/ourselves to sacrifice quality for quantity or by scaling up the number of people doing work by accepting lower quality?
My current thinking:
1. There are clear, simple, robust-seeming arguments for why more quantity* is desirable, in far mode.
2. Deference to more senior EAs seems to point pretty heavily towards focusing on quality over quantity.
3. When I look at specific interventions/grant-making opportunities in near mode, I'm less convinced they are a good idea, and lean towards earlier high-quality work is necessary before scaling.
The conflict between the very different levels of considerations in #1 vs #2 and #3 makes me fairly confused about where the imbalance is, but still maybe worth considering further given just how huge a problem a potential imbalance could be (in either direction).
*Note that there was a bit of slippage in my phrasing, while at the frontiers there's a clear quantity vs average quality tradeoff at the output level, the function that translates inp... (read more)
Something that I think is useful to keep in mind as you probe your own position, whether by yourself, or in debate with others, is:
what's the least surprising piece of evidence, or set of evidence, that would be enough for me to change my mind?
I think sometimes I e.g. have a object-level disagreement with someone about a technology or a meta-disagreement about the direction of EA strategy, and my interlocutor says something like "oh I'll only change my mind if you demonstrate that my entire understanding of causality is wrong or everything I learned in the... (read more)
The General Longtermism team at Rethink Priorities is interested in generating, fleshing out, prioritizing, and incubating longtermist megaprojects.
But what are longtermist megaprojects? In my mind, there are tentatively 4 criteria:
Scale of outcomes: The outputs should be large in scope, from a longtermist point of view. A project at scale should have a decent shot of reducing probability of existential risk by a large amount. Perhaps we believe that after considerable research, we have a decent chance of concluding that the project will reduce existential risk by >0.01% (a "basis point").*
Cost-effectiveness: The project should have reasonable cost-effectiveness. Given limited resources, we shouldn't spend all of them on a project that cost a lot of money and human capital for merely moderate gain. My guess is that we ought to have a numerical threshold of between 100M and 3B (best guess for current target: 500M) for financial plus human capital cost per basis point of existential risk averted.
Scale of inputs: The inputs should be fairly large as well. An extremely impressive paper or a single conversation with the right person, no matter how impactful, should not count as
Meta: It seems to me that the EA community talks about "red teaming" our own work a lot more often than they did half a year ago. It's unclear to me how much my ownshortforms instigated this, vs. e.g. independent convergence.
This seems like a mildly important thing for me to track, as it seems important to me to gauge what fraction of my simple but original-ish ideas are only counterfactually a few months "ahead of the curve," vs genuinely novel and useful for longer.
Whenever someone asked the Comet King why he took the weight of the whole world on his shoulders, he’d just said “Somebody has to and no one else will.” (Source 2)
4.
If some pandemic breaks out that humanity isn’t ready for, mother nature isn’t going to say “well you guys gave it a good shot, so I will suspend the laws of biology for now” – we’re just all going to be dead. Consequentialist ethics is about accepting that fact – accepting that “trying really hard” doesn’t count for anything. (Source)
I think "Reality doesn’t grade on a curve" might originally be from Scott
Alexander's Transhumanist Fables
[https://www.lesswrong.com/s/XsMTxdQ6fprAQMoKi/p/AYbhqi65SWzHzy7Xx].
If I explicitly disagreed with a subpoint in your post/comment, you should assume that I'm only disagreeing with that subpoint; you should NOT assume that I disagree with the rest of the comment and are only being polite. Similarly, if I reply with disagreement to a comment or post overall, you should NOT assume I disagree with your other comments or posts, and certainly I'm almost never trying to admonish you as a person. Conversely, agreements with subpoints should not be treated as agreements with your overall point, agreements with the overall point of an article should not be treated as an endorsement of your actions/your organization, and so forth.
I welcome both public and private feedback on my own comments and posts, especially points that note if I say untrue things. I try to only say true things, but we all mess up sometimes. I expect to mess up in this regard more often than most people, because I'm more public with my output than most people.
Would you be interested in writing a joint essay or perspectives on being Asian American, maybe taking any number of angles, that you can decide upon (e.g. inside liminal spaces of this identity, inside subcultures of tech or EA culture?).
Something tells me you have useful opinions on this topic, and that these are different to mine.
Adding background for context/calibration
This might help you decide whether you want to engage:
I personally have had good experiences or maybe I am blind and unduly privileged. This is basically what I’ll write. I’m OK if my content is small or a minority of the writing.
I personally don’t use the approaches today’s activists use. Instead, I see activism as instrumental to EA cause areas. (Another way of looking at this is that if we thought it was the right thing to do, we could promote the relevant activism/causes to EA causes.)
Based on the two points above, I do not want privilege or attention as a result of the essay or related activity (but I am happy if others seek it).
Something that needs attention are efforts to pool Asian’s with Caucasians, as to cut off Asian (or at least Han Chinese) status as a distinct group with valid opinions o
This seems interesting but I don't currently think it'd trade off favorably against EA work time. Some very quickly typed thoughts re: personal experiences (apologies if it sounds whiny).
1. I've faced minor forms of explicit racism and microaggressions, but in aggregate they're pretty small, possibly smaller than the benefit of speaking a second language and cultural associations.
2. I expect my life would be noticeably better if I were a demographically twin who's Caucasian. But I'm not confident about this.
3. This is almost entirely due to various subtle forms of discrimination at a statistical level, rather than any forms of outright aggression or discrimination.
3a) e.g. I didn't get admitted to elite or even top 50 colleges back in 2011 when the competition was much less stiff than now. I had 1570/1600 SAT, 7 APs (iirc mostly but not all 5s), essays that won minor state-level awards, etc. To be clear, I don't think my profile was amazing or anything but statistically I think my odds would've been noticeably higher if I weren't Asian. OTOH I'm not confident that elite college attendance does much for someone controlling for competence (I think mostly the costs are soc... (read more)
To put concrete numbers into this, I think I will trade 25% of my counterfactual
lifetime consumption if I keep my memories, experiences, personality, abilities,
etc intact, but will a) be implicitly read to others as Caucasian unless they
explicitly ask, and b) I have a relationship with my family that's close to that
of a median EA. However, I will not be willing to trade 80%.
I suspect this is more than most people of my ethnicity would be willing to
trade (and if anything most people won't make this trade even if you pay them to
do so).
(Note that these numbers sound higher than they actually are because it's harder
to trade money for happiness than most people think; also I specified %
consumption rather than $s because there's a ~logarithmic relationship between
money and happiness).
This is entirely thinking in terms of happiness/selfish goals. I'm pretty
confused about what it means for impact. Like naively I don't see clearly better
options than my current job if I was the same person but a different ethnicity,
but clearly there are more options that will open up (and a few that will close)
and because impact is so heavy-tailed, anything from -0.3x more impact to 3x
seems not-crazy to me.
2
Charles He
1y
Disclaimer: My original goal was to open up a thread relevant for many EAs, and
understand the world better. I think your content is interesting, but some
aspects of this discussion that I am continuing are (way) more personal than I
intended—and maybe discussion is at your personal cost, so feel free not to
answer.
This seems really important to you.
From a cold reading of the text, my initial impression is that a) and b) are
really different in nature. I guess maybe you are implying the situation with b)
is related to with your family background and choices as an EA.
That seems valid and important to be aware of. I'm not immediately sure I
understand if this is a systemic issue to people with our background, and maybe
that is my ignorance.
Those scaling factors seem big to me, and suggest a defect that should be
addressed (although there's many other constituencies or needs for attention in
EA).
(Again it is possible I am a traitor but) for myself, I don't immediately see
access to any "leadership opportunity" that is hampered by my background. I
suspect the opposite is true. Also, there seems to be several junior EAs moving
into positions we might consider senior, and this is plausibly related to their
background (in addition to working in the relevant country).
If what you are saying is true, this seems like it should be addressed.
I think that a consideration is that, because of the leadership/funding
situation in EA, I expect that you (and everyone else) is basically furiously
tapping the shoulders of every trusted, talented person who can plausibly get
into leadership, all the time. Maybe Asians are being left out somehow, which is
possible, but needs a clear story.
8
Linch
1y
I mean to be clear I don't know if I'm right! But for the relatively offhand
estimate, I'm mostly thinking about career options outside of institutional EA.
E.g. politics or media. I do basically think things like looks and accent matter
enough to give you something like a 2-3x factor change in probability x
magnitude of success from race alone, probably more (I haven't looked at the
stats closely but I suspect you empirically see roughly this level of
underrepresentation of Asian men in politics and media, when naively based on
other characteristics like educational level or wealth you'd expect higher
representation).
Likewise for the -.3x I was thinking about losing the option value of doing
stuff in China, rather than things within institutional EA.
I do agree that it's be plausible that it's easier to advance as an ethnic
minority within EA, which cuts against my general point about impact. Main
consideration against is that pro-Asian discrimination will look and feel
explicit (e.g. "we don't want 10 white people in the front page of our AI
company," "we need someone who speaks Chinese for our AI
governance/international cooperation research to go better" ), whereas
anti-Asian discrimination will be pretty subtle and most people won't even read
it as such (e.g. judging how much people "vibe" with you in conversations/at
parties to select cofounders over e.g. online output or past work performance,
relying on academic credentials that are implicitly racist over direct cognitive
or personality tests for hiring). But I do think it's more likely that the net
effect within EA is biased in my favor rather than against, with high
uncertainty.
Thanks for writing this, this is thoughtful, interesting and rich in content. I think others benefitted from reading this too.
Also, a reason I asked was that I was worried about the chance that I was ignorant about Asian-American experiences for idiosyncratic reasons. The information you provided was useful.
There is other content you mentioned that seems important (3c and 6). I will send a PM related to this. Maybe there are others reading who know you who also would like to listen to your experiences on these topics.
Re the post On Elitism in EA. Here is the longer version of my thoughts before I realized it could be condensed a lot:
I don't think I follow your model. You define elitism the following way:
Elitism in EA usually manifests as a strong preference for hiring and funding people from top universities, companies, and other institutions where social power, competence, and wealth tend to concentrate. Although elitism can take many other forms, for our purposes, we’ll be using this definition moving forward.
In other words, the "elite" is defined as people who... (read more)
Cute theoretical argument for #flattenthecurve at any point in the distribution
What is #flattenthecurve?
The primary theory behind #flattenthecurve is assuming that everybody who will get COVID-19 will eventually get it anyway...is there anything else you can do?
It turns out it’s very valuable to
Delay the spread so that a) the peak of the epidemic spread is lower (#flattenthecurve)
Also to give public health professionals, healthcare sy
Should there be a new EA book, written by somebody both trusted by the community and (less importantly) potentially externally respected/camera-friendly?
I think the 80,000 hours and EA handbooks were maybe trying to do this, but for whatever reason didn't get a lot of traction?
I suspect that the issue is something like not having a sufficiently strong "voice"/editorial line, and what you want for a book that's a)bestselling and b) does not sacrifice nuance too much is one final author + 1-3 RAs/ghostwriters.
Does the Precipice count? And I think Will Macaskill is writing a new book.
But I have the vague sense that public-facing books may be good for academics'
careers anyway. Evidence for this intuition:
(1) Where EA academics have written them, they seem to be more highly cited than
a lot of their other publications, so the impact isn't just "the public" (see
Google Scholar pages for Will Macaskill, Toby Ord, Nick Bostrom, Jacy Reese
Anthis -- and let me know if there are others who have written public-facing
books! Peter Singer would count but has no Google Scholar page)
(2) this article about the impact of Wikipedia. It's not about public-facing
books but fits into my general sense that "widely viewed summary content
by/about academics can influence other academics"
https://conference.druid.dk/acc_papers/2862e909vshtezgl6d67z0609i5bk6.pdf
[https://conference.druid.dk/acc_papers/2862e909vshtezgl6d67z0609i5bk6.pdf]
Plus all the usual stuff about high fidelity idea transmission being good.
So yes, more EA books would be good?
2
Linch
2y
I think The Precipice is good, both directly and as a way to communicate a
subsection of EA thought, but EA thought is not predicated on a high probability
of existential risk, and the nuance might be lost on readers if The Precipice
becomes the default "intro to EA" book.
What will a company/organization that has a really important secondary mandate to focus on general career development of employees actually look like? How would trainings be structured, what would growth trajectories look like, etc?
When I was at Google, I got the distinct impression that while "career development" and "growth" were common buzzwords, most of the actual programs on offer were more focused on employee satisfaction/retention than growth. (For example, I've essentially never gotten any feedback on my selection of training courses or books that I bought with company money, which at the time I thought was awesome flexibility, but in retrospect was not a great sign of caring about growth on the part of the company).
Edit: Upon a reread I should mention that there are other ways for employees to grow within the company, eg by having some degree of autonomy over what projects they want to work on.
I think there are theoretical reasons for employee career growth being underinvested by default. Namely, that the costs of career growth are borne approximately equally between the employer and the employee (obviously this varies from case to case), whil... (read more)
Definitely agreed. That said, I think some of this should probably be looked
through the lens of "Should EA as a whole help people with personal/career
development rather than specific organizations, as the benefits will accrue to
the larger community (especially if people only stay at orgs for a few years).
I'm personally in favor of expensive resources being granted to help people
early in their careers. You can also see some of this in what OpenPhil/FHI
funds; there's a big focus on helping people get useful PhDs. (though this helps
a small minority of the entire EA movement)
Edit: By figuring out ethics I mean both right and wrong in the abstract but also what the world empirically looks like so you know what is right and wrong in the particulars of a situation, with an emphasis on the latter.
I think a lot about ethics. Specifically, I think a lot about "how do I take the best action (morally), given the set of resources (including information) and constraints (including motivation) that I have." I understand that in philosophical terminology this is only a small subsection of applied ethics, and yet I spend a lot of time thinking about it.
One thing I learned from my involvement in EA for some years is that ethics is hard. Specifically, I think ethics is hard in the way that researching a difficult question or maintaining a complicated relationship or raising a child well is hard, rather than hard in the way that regularly going to the gym is hard.
When I first got introduced to EA, I believed almost the opposite (this article presents something close to my past views well): that the hardness of living ethically is a matter of execution and will, rather than that of constantly making tradeoffs in a difficult-to-navigate domain.
I'm at a good resting point in my current projects, so I'd like to take some time off to decide on "ambitious* projects Linch should be doing next," whether at RP or elsewhere.
Excited to call with people who have pitches, or who just want to be a sounding board to geek out with me.
*My current filter on "ambition" is “only consider projects with a moral value >> that of adding 100M to Open Phil’s coffers assuming everything goes well.” I'm open to arguments that this is insufficiently ambitious, too ambitious, or carving up the problem at the wrong level of abstraction.
One alternative framing is thinking of outputs rather than intermediate goals, eg, "only consider projects that can reduce x-risk by >0.01% assuming everything goes well."
For the sake of clarification, I do think it's more likely than not that I'll
stay at Rethink Priorities for quite some time, even if I do end up incubating a
new project (though of course details depend on organizational needs etc). I
mention this because I talked to multiple people who thought that I was more
likely to leave than stay, and in some cases made additional inferences about
the quality or importance of research there based on pretty little information.
(I'm still fairly confused about this implication and this isn't the first time
that a job-related post I made
[https://www.facebook.com/linchuan.zhang/posts/2404989282925175?comment_id=2406635772760526&__cft__[0]=AZVrwNlM88C0Dmf1mKsxEdcOAt9_rBpLCn2YPdMRYG3-4k1LKrLNfvv5ktJrnnz9bZ-_ZXR3gMgIz5vlNCR30JSKUYwY-OzOA1IsIhdcsvTgbw&__tn__=R]-R]
apparently had the wrong connotation (I left Google on good terms to join a
startup, some people emailed me being worried that I was laid off). My best
guess here is that people are usually just really reluctant to be candid about
their career-change-related considerations so there's more of a euphemism
treadmill here than elsewhere).
5
MichaelStJules
2y
It's hard to know what adding an extra $100M to Open Phil would do, since
they're already having a hard time spending more on things, especially
x-risk-related stuff (which also presumably has support from Sam Bankman-Fried,
with a net worth of >$20B, but maybe only a small share is liquid).
I think a better framing might be projects that Open Phil and other funders
would be inclined to fund at ~$X (for some large X, not necessarily 100M), and
have cost-effectiveness similar to their current last dollar in the relevant
causes or better.
It seems smaller than megaprojects
[https://forum.effectivealtruism.org/posts/ckcoSe3CS2n3BW3aT/what-ea-projects-could-grow-to-become-megaprojects]
($100M/year, not $100M in total).
If you wanted to do something similar to adding $100M to Open Phil, you could
look into how to get them to invest better or reduce taxes. $100M is <0.5% of
Dustin Moskovitz's/Open Phil's wealth, and I think 0.5% higher market returns is
doable (a lame answer that increases risk is to use a tiny bit of leverage or
hold less in bonds, but there may be non-risk-increasing ways to increase
returns, e.g. sell FB stock and invest more broadly).
2
Linch
2y
Michael Dickens has already done a bunch of work on this, and I don't feel
confident in my ability to improve on it, especially given that, to the best of
my knowledge, Michael D's advice are currently not followed by the super-HNWs
(for reasons that are not super-legible to me, though admittedly I haven't
looked too deeply).
I agree that this might be a better framing. I feel slightly queasy about it as
it feels a bit like "gaming" to me, not sure.
4
MichaelDickens
2y
I don't really know, but my guess is it's mostly because of two things:
1. Most people are not strategic and don't do cost-benefit analyses on big
decisions. HNW people are often better at this than most, but still not
great.
2. On the whole, investment advisors are surprisingly incompetent. That raises
the question of why this is. I'm not sure, but I think it's mainly
principal-agent problems—they're not incentivized to actually make money for
their clients, but to sounds like they know what they're talking about so
they get hired. And the people who evaluate investment advisors know less
than the advisors do (almost by definition), so aren't good at evaluating
them.
I just wrote a longer essay about this:
https://mdickens.me/2021/10/29/obvious_investing_facts/
[https://mdickens.me/2021/10/29/obvious_investing_facts/] (I had been thinking
about this concept for a while, but your comment motivated me to sit down and
write it.)
There is a reasonable third possibility, which is that nobody implements my
unorthodox investing ideas because I'm wrong. I believe it would be valuable for
a competent person with relevant expertise to think about the same things I've
thought about, and see if they come up with different results.
ETA: One counterexample is Bill Gates. One of my pet issues is that people
concentrate their wealth too much
[https://forum.effectivealtruism.org/posts/TXgrYjW2D9JbgbTk7/the-risk-of-concentrating-wealth-in-a-single-asset].
But Bill Gates diversified away from Microsoft fairly quickly, and right now
only a pretty small % of his wealth is still in Microsoft. I don't know how he
pulled that off without crashing the stock, but apparently it's possible.
1
Daniel_Eth
2y
Can you link to this work?
3
Linch
2y
Hmm most of his EAForum output
[https://forum.effectivealtruism.org/users/michaeldickens] looks like this.
These posts may be especially important/salient:
https://forum.effectivealtruism.org/posts/g4oGNGwAoDwyMAJSB/how-much-leverage-should-altruists-use
[https://forum.effectivealtruism.org/posts/g4oGNGwAoDwyMAJSB/how-much-leverage-should-altruists-use]
https://forum.effectivealtruism.org/posts/TXgrYjW2D9JbgbTk7/the-risk-of-concentrating-wealth-in-a-single-asset
[https://forum.effectivealtruism.org/posts/TXgrYjW2D9JbgbTk7/the-risk-of-concentrating-wealth-in-a-single-asset]
https://forum.effectivealtruism.org/posts/zKFcC87iZXvk7yhbP/uncorrelated-investments-for-altruists
[https://forum.effectivealtruism.org/posts/zKFcC87iZXvk7yhbP/uncorrelated-investments-for-altruists]
3
Daniel_Eth
2y
I think I disagree and would prefer Linch's original idea; there may be things
that are much more cost-effective than OPP's current last dollar (to the point
that they'd provide >>$100M of value for <<$100M to OPP), but which can't absorb
$X (or which OPP wouldn't pay $X for, due to other reasons).
2
MichaelStJules
2y
I think you can adjust my proposal for this:
1. Cost-effectiveness similar to or better than Open Phil's last dollar.
2. Impact similar or better than the last $100 million Open Phil spends.
Maybe having a single number is preferable. Ben Todd recommended going for the
project with the highest potential impact
[https://forum.effectivealtruism.org/posts/Kwj6TENxsNhgSzwvD/most-research-advocacy-charities-are-not-scalable?commentId=7yXM9ovMw3uKR2zy7#7yXM9ovMw3uKR2zy7]
(with a budget constraint).
A corollary of background EA beliefs is that everything we do is incredibly important.
This is covered elsewhere in the forum, but I think an important corollary of many background EA + longtermist beliefs is that everything we do is (on an absolute scale) very important, rather than useless.
I know some EAs who are dispirited because they donate a few thousand dollars a year when other EAs are able to donate millions. So on a relative scale, this makes sense -- other people are able to achieve >1000x the impact through their donations as you do.
But the "correct" framing (I claim) would look at the absolute scale, and consider stuff like we are a) among the first 100 billion or so people and we hope there will one day be quadrillions b) (most) EAs are unusually well-placed within this already very privileged set and c) within that even smaller subset again, we try unusually hard to have a long term impact, so that also counts for something.
EA genuinely needs to prioritize very limited resources (including time and attention), and some of the messages that radiate from our community, particularly around relative impact of different people, may come across as... (read more)
Can you spell out the impact estimation you are doing in more detail? It seems
to me that you first estimate how much a janitor at an org might impact the
research productivity of that org, and then there's some multiplication related
to the (entire?) value of the far future. Are you assuming that AI will
essentially solve all issues and lead to positive space colonization, or
something along those lines?
2
Linch
2y
I think the world either ends or some other form of (implied permanent) x-risk
in the next 100 years or it doesn't. And if the world doesn't end in the next
100 years, we eventually will either a) settle the stars or b) ends or
drastically curtails at some point >100 years out.
I guess I assume b) is pretty low probability with AI, like much less than 99%
chance. And 2 orders of magnitude isn't much when all the other numbers are
pretty fuzzy and spans that many orders of magnitude.
(A lot of this is pretty fuzzy).
2[anonymous]2y
So is the basic idea that transformative AI not ending in an existential
catastrophe is the major bottleneck on a vastly positive future for humanity?
2
Linch
2y
No, weaker claim than that, just saying that P(we spread to the stars|we don't
all die or are otherwise curtailed from AI in the next 100 years) > 1%.
(I should figure out my actual probabilities on AI and existential risk with at
least moderate rigor at some point, but I've never actually done this so far).
2[anonymous]2y
Thanks. Going back to your original impact estimate, I think the bigger
difficulty I have in swallowing your impact estimate and claims related to it
(e.g. "the ultimate weight of small decisions you make is measured not in
dollars or relative status, but in stars") is not the probabilities of AI or
space expansion, but what seems to me to be a pretty big jump from the potential
stakes of a cause area or value possible in the future without any existential
catastrophes, to the impact that researchers working on that cause area might
have.
2
Linch
2y
Can you be less abstract and point, quantitatively, to which numbers I gave seem
vastly off to you and insert your own numbers? I definitely think my numbers are
pretty fuzzy but I'd like to see different ones before just arguing verbally
instead.
(Also I think my actual original argument was a conditional claim, so it feels a
little bit weird to be challenged on the premises of them! :)).
I don't know, but my best guess is that "janitor at MIRI"-type examples reinforce a certain vibe people don't like — the notion that even "lower-status" jobs at certain orgs are in some way elevated compared to other jobs, and the implication (however unintended) that someone should be happy to drop some more fulfilling/interesting job outside of EA to become MIRI's janitor (if they'd be good).
I think your example would hold for someone donating a few hundred dollars to MIRI (which buys roughly 10^-4 additional researchers), without triggering the same ideas. Same goes for "contributing three useful LessWrong comments on posts about AI", "giving Superintelligence to one friend", etc. These examples are nice in that they also work for people who don't want to live in the Bay, are happy in their current jobs, etc.
Anyway, that's just a guess, which doubles as a critique of the shortform post. But I did upvote the post, because I liked this bit:
But the "correct" framing (I claim) would look at the absolute scale, and consider stuff like we are a) among the first 100 billion or so people and we hope there will one day be quadrillions b) (most) EAs are unusually well-placed within this already very privileged set and c) within that even smaller subset again, we try unusually hard to have a long term impact, so that also counts for something.
I agree that the vibe you're describing tends to be a bit cultish precisely
because people take it too far. That said, it seems right that low prestige jobs
within crucially needed teams can be more impactful than high-prestige jobs
further away from the action. (I'm making a general point; I'm not saying that
MIRI is necessarily a great example for "where things matter," nor am I saying
the opposite.) In particular, personal assistant strikes me as an example of a
highly impactful role (because it requires a hard-to-replace skillset).
(Edit: I don't expect you to necessarily disagree with any of that, since you
were just giving a plausible explanation for why the comment above may have
turned off some people.)
2
Linch
2y
I agree with this, and also I did try emphasizing that I was only using MIRI as
an example. Do you think the post would be better if I replaced MIRI with a
hypothetical example? The problem with that is that then the differences would
be less visceral.
6
Linch
2y
FWIW I'm also skeptical
[https://forum.effectivealtruism.org/posts/myp9Y9qJnpEEWhJF9/shortform?commentId=9LAYrb8RCqtWHPkbn]
of naive ex ante differences of >~2 orders of magnitude between causes, after
accounting for meta-EA effects. That said, I also think maybe our culture will
be better if we celebrate doing naively good things over doing things that are
externally high status.*
But I don't feel too strongly, main point of the shortform was just that I talk
to some people who are disillusioned because they feel like EA tells them that
their jobs are less important than other jobs, and I'm just like, whoa, that's
just such a weird impression on an absolute scale (like knowing that you won a
million dollars in a lottery but being sad that your friend won a billion). I'll
think about how to reframe the post so it's less likely to invite such relative
comparisons, but I also think denying the importance of the relative comparisons
is the point.
*I also do somewhat buy arguments by you and Holden Karnofsky and others that
it's more important for skill/career capital etc building to try to do really
hard things even if they're naively useless. The phrase "mixed strategy" comes
to mind.
2
Habryka
2y
I feel like the meta effects are likely to exaggerate the differences, not
reduce them? Surprised about the line of reasoning here.
2
Linch
2y
Hmm I think the most likely way downside stuff will happen is by flipping the
sign rather than reducing the magnitude, curious why your model is different.
I wrote a bit more in the linked
[https://forum.effectivealtruism.org/posts/myp9Y9qJnpEEWhJF9/shortform?commentId=9LAYrb8RCqtWHPkbn]
shortform.
2
Aaron Gertler
2y
This is a reasonable theory. But I think there are lots of naively good things
that are broadly accessible to people in a way that "janitor at MIRI" isn't,
hence my critique.
(Not that this one Shortform post is doing anything wrong on its own — I just
hear this kind of example used too often relative to examples like the ones I
mentioned, including in this popular post
[https://forum.effectivealtruism.org/posts/cCrHv4Rn3StCXqgEw/in-praise-of-unhistoric-heroism],
though the "sweep the floors at CEA" example was a bit less central there.)
I wasn't sure where to add this comment, or idea rather. But I have something to propose hair that I think would make a gigantic impact on the well-being of millions. I don't have a fancy research or such to back it up yet but I'm sure there's money out there to support if needed. So here goes ... I remember learning about ancient times and Greek mythology and such and was very fascinated with the time period and a lot of it's heroes. I do believe that's back then four wars or sometimes fought by having two of the best warriors battled and place of whole a... (read more)
Hi, I think this is an interesting idea, thanks for raising it. People have
raised this a few times before, mostly in science fiction. You might wonder why
countries don't do this already.
The main issue is lack of enforcement mechanism. If someone does well with their
videogame armies, the loser could agree to abide by the rules. Or they can
"cheat" and just invade your country with a real army anyway. And there are
strong incentives for cheating!
I think this idea is theoretically workable but it's pretty hard to get it to
work. You probably need an external enforcement mechanism somehow.
This paper [
https://web.stanford.edu/group/fearon-research/cgi-bin/wordpress/wp-content/uploads/2013/10/Rationalist-Explanations-for-War.pdf
] has a bunch of jargon but might be related.
2[anonymous]5mo
The theoretically workable version of this idea just seems like diplomacy and
international relations.
If decision-makers of both nations agree about all possible outcomes of a war
between the two nations and their likelihoods, its easier for them to avoid
engaging in war and pick trade, military and other policies that reflect these
likelihoods instead. (Whichever nation is more likely to "win" the war would be
able to negotiate better deals.)
If both decision-makers disagree, it seems useful for them (or their
representatives/diplomats/etc) to get into the same room and actually simulate
what a war would look like, figure out where they disagree on potential outcomes
and then try to debate with a view to agree. Obviously there are incentives for
both sides to misrepresent their actual beliefs in the conversation, but ideally
whatever conversation happens influences the actual beliefs of the
decision-makers and pushes these beliefs closer to each other as well as closer
to the Truth. Which in turn makes it easier to avoid war.
2
Linch
4mo
I thought the Fearon paper was pretty good. Have you read it by any chance?
4[anonymous]4mo
One point the paper misses is precommitments to not be peaceful or not trade.
Leaders may have incentives to commit to "being a certain type of person" and
signaling and adopting certain personality traits, even if in later situations
making this commitment causes problems for them. Humans often make such
precommitments because humans have partial but not full ability to verify each
other's behaviours and thinking. (This makes humans slightly different from game
theoretic agents who are usually assumed fully black box or fully white box.)
For instance a mafia could precommit to always retaliating if any of their
members is killed by a rival gang. This could reduce likelihood they get
attacked in the first place, but when they do get attacked they have now
committed to full retaliation and cannot choose to negotiate a trade instead.
Another example, with nuclear war, leaders may need to precommit to be the kind
of person who will always retaliate with a second strike (if they face a first
strike), even though a first strike could already make civilisation sufficiently
worse off that a second strike is just needless harm that won't achieve any
benefit (after the first strike has already been made).
See also: commitment races
https://www.lesswrong.com/posts/brXr7PJ2W4Na2EW2q/the-commitment-races-problem
[https://www.lesswrong.com/posts/brXr7PJ2W4Na2EW2q/the-commitment-races-problem]
4[anonymous]4mo
Thank you for linking it and sorry I didn't notice it the first time! It
overlaps almost entirely with how I was thinking about the topic.
4
Linch
4mo
I'm glad you liked the paper! And no worries on not noticing it the first time;
I don't read every paper people on the forum link before commenting, and I
certainly don't expect other people to.
2
Ramiro
4mo
Oh I was hoping you would propose this:
https://www.smbc-comics.com/comic/the-end-of-history
[https://www.smbc-comics.com/comic/the-end-of-history]
Sorry for the joke, I actually like your idea. But the military indeed sorta
prevent having wars by doing military exercises to expensively signal strength
and capabilities. That's how we have prevented WW III so far. So the crux is
doing this without such an economic waste.
1
Lavender West
6mo
Please excuse any type of a mess of this post. Not sure if anybody's even going
to read it but it wasn't edited before I hit the submit button and I literally
have just been rambling into the speak to text on my phone, which at times seems
to hear things other than once I am trying to have it relay
2
Linch
6mo
Thanks for the comment! You might find this guide to the forum
[https://forum.effectivealtruism.org/about] helpful.
Here are some things I've learned from spending the better part of the last 6 months either forecasting or thinking about forecasting, with an eye towards beliefs that I expect to be fairly generalizable to other endeavors.
Note that I assume that anybody reading this already has familiarity with Phillip Tetlock's work on (super)forecasting, particularly Tetlock's 10 commandments for aspiring superforecasters.
1. Forming (good) outside views is often hard but not impossible. I think there is a common belief/framing in EA and rationalist circles that coming up with outside views is easy, and the real difficulty is a) originality in inside views, and also b) a debate of how much to trust outside views vs inside views.
I think this is directionally true (original thought is harder than synthesizing existing views) but it hides a lot of the details. It's often quite difficult to come up with and balance good outside views that are applicable to a situation. See Manheim and Muelhauser for some discussions of this.
2. For novel out-of-distribution situations, "normal" people often trust centralized data/ontologies more than is warranted. See here for a discu... (read more)
Consider making this a top-level post! That way, I can give it the "Forecasting" tag so that people will find it more often later, which would make me happy, because I like this post.
Thanks for the encouragement and suggestion! Do you have recommendations for a
really good title?
2
Aaron Gertler
3y
Titles aren't my forte. I'd keep it simple. "Lessons learned from six months of
forecasting" or "What I learned after X hours of forecasting" (where "X" is an
estimate of how much time you spent over six months).
Recently I was asked for tips on how to be less captured by motivated reasoning and related biases, a goal/quest I've slowly made progress on for the last 6+ years. I don't think I'm very good at this, but I do think I'm likely above average, and it's also something I aspire to be better at. So here is a non-exhaustive and somewhat overlapping list of things that I think are helpful:
Treat this as a serious problem that needs active effort.
In general, try to be more of a "scout" and less of a soldier/zealot.
As much as possible, try to treat ideas/words/stats as tools to aid your perception, not weapons to advance your cause or "win" arguments.
I've heard good things about this book by Julia Galef, but have not read it
Relatedly, have a mental check and try to notice when you are being emotional and in danger of succumbing to motivated reasoning. Always try to think as coherently as you can if possible, and acknowledge it (preferably publicly) when you notice you're not.
I've previously shared this post on CEA's social media and (I think) in an
edition of the Forum Digest. I think it's really good, and I'd love to see it be
a top-level post so that more people end up seeing it, it can be tagged, etc.
Would you be interested in creating a full post for it? (I don't think you'd
have to make any changes — this still deserves to be read widely as-is.)
Is anybody trying to model/think about what actions we can do that are differentially leveraged during/in case of nuclear war, or the threat of nuclear war?
In the early days of covid, most of us were worried early on, many of us had reasonable forecasts, many of us did stuff like buy hand sanitizers and warn our friends, very few of us shorted airline stocks or lobbied for border closures or did other things that could've gotten us differential influence or impact from covid.
Very bad societal breakdown scenarios
There's a spectrum here, but thinking about the more bad scenarios, the
aftermath could be really chaotic, e.g. safety, crime, wouldn't just be an issue
but something like "warlords" might exist. You might have some sort of nominal
law/government, but in practice, violence and coercion would be normal [1].
I think the experiences with hurricane Katrina's breakdown is a good example,
but this would be more severe. Chaos in Libya and other nations are other
examples.
This seems cartoonish, but in this situation, the most impactful thing for the
average EA to do is to involve themselves and gain position in some level of
"government", probably in a nontraditional sense[2].
Strong generalist skills, work ethic, communication, charisma, leadership,
interpersonal judgement, as well as physical endurance would be important. This
would be valuable because I think success, even local, might give access to
larger political/power.
If this seems silly, the alternatives seem sillier? EAs trying to provision
services (farming) or writing papers/lobbying without knowledge of the new
context doesn't seem useful.
1. ^
Because I think some EAs are sensitive, I am trying to not show this, but I
think movies like Threads or The Day After communicate the scenarios I'm
thinking about.
https://en.wikipedia.org/wiki/Threads_(1984_film)
[https://en.wikipedia.org/wiki/Threads_(1984_film)]
https://en.wikipedia.org/wiki/The_Day_After
[https://en.wikipedia.org/wiki/The_Day_After]
2. ^
I think some EAs with a lot of access/connections might get positions in the
post-event central government, but this seems limited to those very
lucky/senior (most government employees will be laid off).
I’d be curious to know: what do you most disagree with the Effective Altruism philosophy, worldview or community about?
I had a response that a few people I respect thought was interesting, so I'm reposting it here:
Insufficient sense of heroic responsibility. "Reality doesn't grade on a curve." It doesn't matter (that much) whether EA did or did not get more things "right" about covid than conventional experts, it matters that millions of people died, and we're still not prepared enough for the next (bigger) pandemic. (similar story in the other cause areas).
Not enough modeling/Fermi/backchaining/coming up with concrete Theories of Change/Victory in decision-guiding ways.
Too much time spent responding to dumb criticisms, insufficient time spent seeking out stronger criticisms.
Overly deferential (especially among the junior EAs) to the larger players. See Ozzie Gooen: "I run into a bunch of people who assume that the EA core is some type of agentic super-brain that makes all moves intentionally. So if something weird is going on, it must be for some eccentric reason of perfect wisdom. " https://www.facebook.com/ozzie.gooen/posts/10165633038585363
A skill/attitude I feel like I improved a lot on in the last year, and especially in the last 3 months, is continuously asking myself whether any work-related activity or research direction/project I'm considering has a clear and genuine/unmovitated story for having a notable positive impact on the long-term future (especially via reducing x-risks), and why.
Despite its simplicity, this appears to be a surprisingly useful and rare question. I think I recommend more people, especially longtermism researchers and adjacent folks, to consider this explicitly, on a regular/almost instinctual basis.
Are there any EAA researchers carefully tracking the potential of huge cost-effectiveness gains in the ag industry from genetic engineering advances of factory farmed animals? Or (less plausibly) advances from better knowledge/practice/lore from classical artificial selection? As someone pretty far away from the field, a priori the massive gains made in biology/genetics in the last few decades seems like something that we plausibly have not priced in in. So it'd be sad if EAAs get blindsided by animal meat becoming a lot cheaper in the next few decades (if this is indeed viable, which it may not be).
This post
[https://www.overcomingbias.com/2012/12/breeding-happier-livestock-no-futuristic-tech-required.html]
may be of interest, in case you haven't seen it already.
2
Linch
2y
Yep, aware of this! Solid post.
4
MichaelStJules
2y
Besides just extrapolating trends in cost of production/prices, I think the main
things to track would be feed conversion ratios
[https://en.wikipedia.org/wiki/Feed_conversion_ratio] and the possibility of
feeding animals more waste products or otherwise cheaper inputs, since feed is
often the main cost of production. Some FCRs are already < 2 and close to 1,
e.g. it takes less than 2kg of input to get 1kg of animal product (this could be
measured in just weight, calories, protein weight, etc..), e.g. for chickens,
some fishes and some insects.
I keep hearing that animal protein comes from the protein in what animals eat
(but I think there are some exceptions, at least), so this would put a lower
bound of 1 on FCR in protein terms, and there wouldn't be much further to go for
animals close to that.
I think a lower bound of around 1 for weight of feed to weight of animal product
also makes sense, maybe especially if you ignore water in and out.
So, I think chicken meat prices could roughly at most halve again, based on
these theoretical limits, and it's probably much harder to keep pushing.
Companies are also adopting less efficient breeds to meet welfare standards like
the Better Chicken Commitment [https://betterchickencommitment.com/], since
these breeds have really poor welfare due to their accelerated growth.
This might be on Lewis Bollard's radar, since he has written about the cost of
production, prices and more general trends in animal agriculture.
Do people have advice on how to be more emotionally resilient in the face of disaster?
I spent some time this year thinking about things that are likely to be personally bad in the near-future (most salient to me right now is the possibility of a contested election + riots, but this is also applicable to the ongoing Bay Area fires/smoke and to a lesser extent the ongoing pandemic right now, as well as future events like climate disasters and wars). My guess is that, after a modicum of precaution, the direct objective risk isn't very high, but it'll *feel* like a really big deal all the time.
In other words, being perfectly honest about my own personality/emotional capacity, there's a high chance that if the street outside my house is rioting, I just won't be productive at all (even if I did the calculations and the objective risk is relatively low).
So I'm interested in anticipating this phenomenon and building emotional resilience ahead of time so such issues won't affect me as much.
I'm most interested in advice for building emotional resilience for disaster/macro-level setbacks. I think it'd also be useful to build resilience for more personal setbacks (eg career/relationship/impact), but I naively suspect that this is less tractable.
The last newsletter from Spencer Greenberg/Clearer Thinking might be helpful:
https://www.clearerthinking.org/post/2020/10/06/how-resetting-your-psychological-baseline-can-make-your-life-better
[https://www.clearerthinking.org/post/2020/10/06/how-resetting-your-psychological-baseline-can-make-your-life-better]
7
Linch
3y
Wow, reading this was actually surprisingly helpful for some other things I'm
going through. Thanks for the link!
2
Misha_Yagudin
3y
I think it is useful to separately deal with the parts of a disturbing event
over which you have an internal or external locus of control. Let's take a look
at riots:
* An external part is them happening in your country. External locus of control
means that you need to accept the situation. Consider looking into Stoic
literature and exercises (say, negative visualizations) to come to peace with
that possibility.
* An internal part is being exposed to dangers associated with them. Internal
locus of control means that you can take action to mitigate the risks.
Consider having a plan to temporarily move to a likely peaceful area within
your country or to another county.
What are the best arguments for/against the hypothesis that (with ML) slightly superhuman unaligned systems can't recursively self-improve without solving large chunks of the alignment problem?
Like naively, the primary way that we make stronger ML agents is via training a new agent, and I expect this to be true up to the weakly superhuman regime (conditional upon us still doing ML).
Here's the toy example I'm thinking of, at the risk of anthromorphizing too much:Suppose I'm Clippy von Neumann, an ML-trained agent marginally smarter than all humans, but nowhere near stratospheric. I want to turn the universe into paperclips, and I'm worried that those pesky humans will get in my way (eg by creating a stronger AGI, which will probably have different goals because of the orthogonality thesis). I have several tools at my disposal:
Try to invent ingenious mad science stuff to directly kill humans/take over the world
But this is too slow, another AGI might be trained before I can do this
Copy myself a bunch, as much as I can, try to take over the world with many copies.
Maybe too slow? Also might be hard to get enough resources to make more copies
But if I'm just a bunch of numbers in a neural net, this entails doing brain surgery via changing my own weights without accidentally messing up my utility function, and this just seems really hard. [...] maybe some AI risk people thinks this is only slightly superhuman, or even human-level in difficulty?
No, you make a copy of yourself, do brain surgery on the copy, and copy the changes to yourself only if you are happy with the results. Yes, I think recursive improvement in humans would accelerate a ton if we had similar abilities (see also Holden on the impacts of digital people on social science).
What is the likelihood that this is within the power of beings say 10x as
intelligent as we are. It seems very plausible to me that there are three
relevant values here (self-improvement, alignment, intelligence) and it could
just be too hard for the superhuman AI to do.
This should pull doom number down right?
2
Rohin Shah
17d
I think it's within the power of beings equally as intelligent as us (similarly
as mentioned above I think recursive improvement in humans would accelerate if
we had similar abilities).
2
Nathan Young
17d
Wait, you think the reason we can't do brain improvement is because we can't
change the weights of individual neurons?
That seems wrong to me. I think it's because we don't know how the neurons work.
Similarly I'd be surprised if you thought that beings as intelligent as humans
could recursively improve NNs. Cos currently we can't do that, right?
2
Rohin Shah
12d
Did you read the link to Cold Takes above? If so, where do you disagree with it?
(I agree that we'd be able to do even better if we knew how the neurons work.)
Humans can improve NNs? That's what AI capabilities research is?
(It's not "recursive" improvement but I assume you don't care about the
"recursive" part here.)
4
Buck
1y
How do you know whether you're happy with the results?
2
Linch
1y
Okay now I'm back to being confused.
2
Rohin Shah
1y
I agree that's a challenge and I don't have a short answer. The part I don't buy
is that you have to understand the neural net numbers very well in some
"theoretical" sense (i.e. without doing experiments), and that's a blocker for
recursive improvement. I was mostly just responding to that.
That being said, I would be pretty surprised if "you can't tell what
improvements are good" was a major enough blocker that you wouldn't be able to
significantly accelerate recursive improvement. It seems like there are so many
avenues for making progress:
* You can meditate a bunch on how and why you want to stay aligned /
cooperative with other copies of you before taking the snapshot that you run
experiments on.
* You can run a bunch of experiments on unmodified copies to see which parts of
the network are doing what things; then you do brain surgery on the parts
that seem most unrelated to your goals (e.g. maybe you can improve your
logical reasoning skills).
* You can create domain-specific modules that e.g. do really good theorem
proving or play Go really well or whatever, somehow provide the
representations from such modules as an "input" to your mind, and learn to
use those representations yourself, in order to gain superhuman intuitions
about the domain.
* You can notice when you've done some specific skill well, look at what in
your mind was responsible, and 10x the size of the learning update. (In the
specific case where you're still learning through gradient descent, this just
means adapting the learning rate based on your evaluation of how well you
did.) This potentially allows you to learn new "skills" much faster (think
of something like riding a bike, and imagine you could give your brain 10x
the update when you did it right).
It's not so much that I think any of these things in particular will work, it's
more that given how easy it was to generate these, I expect there to be so many
such opportunities, especia
4
Linch
2y
Oh wow thanks that's a really good point and cleared up my confusion!! I never
thought about it that way before.
5
Buck
2y
This argument for the proposition "AI doesn't have an advantage over us at
solving the alignment problem" doesn't work for outer alignment—some goals are
easier to measure than others, and agents that are lucky enough to have
easy-to-measure goals can train AGIs more easily.
1
David Johnston
2y
The world's first slightly superhuman AI might be only slightly superhuman at AI
alignment. Thus if creating it was a suicidal act by the world's leading AI
researchers, it might be suicidal in exactly the same way. In the other hand, if
it has a good grasp of alignment then it's creators might also have a good grasp
of alignment.
In the first scenario (but not the second!), creating more capable but not fully
aligned descendants seems like it must be a stable behaviour of intelligent
agents, as by assumption
1. behaviour of descendants is only weakly controlled by parents
2. the parents keep making better descendants until the descendants are
strongly superhuman
I think that Buck's also right that the world's first superhuman AI might have a
simpler alignment problem to solve.
1
Dan Elton
2y
"It doesn't naively seem like AI risk is noticeably higher or lower if recursive
self-improvement doesn't happen." If I understand right, if recursive
self-improvement is possible, this greatly increases the take-off speed, and
gives us much less time to fix things on the fly. Also, when Yudkowsky has
talked about doomsday foom my recollection is he was generally assuming
recursive self-improvement, of a quite-fast variety. So it is important.
(Implementing the AGI in a Harvard architecture, where source code is not in
accessible/addressable memory, would help a bit prevent recursive self
improvement)
Unfortunately it's very hard to reason about how easy/hard it would be because
we have absolutely no idea what future existentially dangerous AGI will look
like. An agent might be able to add some "plugins" to its source code (for
instance to access various APIs online or run scientific simulation code) but if
AI systems continue trending in the direction they are, a lot of it's
intelligence will probably be impenetrable deep nets.
An alternative scenario would be that intelligence level is directly related to
something like "number of cortical columns" , and so to get smarter you just
scale that up. The cortical columns are just world modeling units, and something
like an RL agent uses them to get reward. In that scenario improving your world
modeling ability by increasing # of cortical columns doesn't really effect
alignment much.
All this is just me talking off the top of my head. I am not aware of this being
written about more rigorously anywhere.
I've been asked to share the following. I have not loved the EA communications from the campaign. However, I do think this is plausibly the most cost-effective use of time this year for a large fraction of American EAs and many people should seriously consider it (or just act on it and reflect later), but I have not vetted these considerations in detail.
[[URGENT]] Seeking people to lead a phone-banking coworking event for Carrick Flynn's campaign today, tomorrow, or Tuesday in gather.town ! There is an EA coworking room in gathertown already. This is a strong counterfactual opportunity! This event can be promoted on a lot of EA fb pages as a casual and fun event (after all, it won't even be affiliated with "EA", but just some people who are into this getting together to do it), hopefully leading to many more phone banker hours in the next couple days.
Would you or anyone else be willing to lead this? You (and other hosts) will be trained in phonebanking and how to train your participants in phonebanking.
Please share with people you think would like to help, and DM Ivy and/or CarolineJ (likely both as Ivy is traveling).
You can read more about Carrick's campaign from an EA... (read more)
While talking to my manager (Peter Hurford), I made a realization that by default when "life" gets in the way (concretely, last week a fair amount of hours were taken up by management training seminars I wanted to attend before I get my first interns, this week I'm losing ~3 hours the covid vaccination appointment and in expectation will lose ~5 more from side effects), research (ie the most important thing on my agenda that I'm explicitly being paid to do) is the first to go. This seems like a bad state of affairs.
I suspect that this is more prominent in me than most people, but also suspect this is normal for others as well. More explicitly, I don't have much "normal" busywork like paperwork or writing grants and I try to limit my life maintenance tasks (of course I don't commute and there's other stuff in that general direction). So all the things I do are either at least moderately useful or entertaining. Eg, EA/work stuff like reviewing/commenting on other's papers, meetings, mentorship stuff, slack messages, reading research and doing research, as well as personal entertainment stuff like social media, memes, videogames etc (which I do much more than I'm willing to admi... (read more)
This seems like a good topic for a call with a coach/coworker because there are
a lot of ways to approach it. One easy-to-implement option comes from 80k's
podcast with Tom Kalil: "the calendar is your friend."
In Tom's case, it was Obama's calendar! I have a more low-key version I use.
When I want to prioritize research or writing, I will schedule a time with my
manager or director to get feedback on a draft of the report I'm writing. It's a
good incentive to make some meaningful progress - hard to get feedback if you
haven't written anything! - and makes it a tiny bit easier to decline or
postpone meetings, but it is still somewhat flexible, which I find really
useful.
4
MaxRa
2y
Yeah, I can also relate a lot (doing my PhD). One thing I noticed is that my
motivational system slowly but surely seems to update on my AI related worries
and that this now and then helps keeping me focused on what I actually think is
more important from the EA perspective. Not sure what you are working on, but
maybe there are some things that come to your mind how to increase your overall
motivation, e.g. by reading or thinking of concrete stories of why the work is
important, and by talking to others why you care about the things you are trying
to achieve.
I hear that this similar to a common problem for many entrepreneurs; they spend much of their time on the urgent/small tasks, and not the really important ones.
One solution recommended by Matt Mochary is to dedicate 2 hours per day of the most productive time to work on the the most important problems.
So framing this in the inverse way – if you have a windfall of time from "life" getting in the way less, you spend that time mostly on the most important work, instead of things like extra meetings. This seems good. Perhaps it would be good to spend less of your time on things like meetings and more on things like research, but (I'd guess) this is true whether or not "life" is getting in the way more.
This is a really good point, I like the reframing.
5
FJehn
2y
This resonated with me a lot. Unfortunately, I do not have a quick fix. However,
what seems to help at least a bit for me is seperating planning for a day and
doing the work. Every workday the last thing I do (or try to do) is look at my
calendar and to do lists and figure out what I should be doing the next day. By
doing this I think I am better at assessing at what is important, as I do not
have to do it at that moment. I only have to think of what my future self will
be capable of doing. When the next day comes and future self turns into present
self I find it really helpful to already having the work for the day planned for
me. I do not have to think about what is important, I just do what past me
decided.
Not sure if this is just an obvious way to do this, but I thought it does not
hurt to write it down.
4
Jamie_Harris
2y
Side question: what was the management training you took, and would you
recommend it?
6
saulius
2y
I think that all of us RP intern managers took the same 12-hour management
training from The Management Center [https://www.managementcenter.org/]. I
thought that there was some high-quality advice in it but I'm not sure if it
applied that much to our situation of managing research interns. I haven't been
to other such trainings so I can't compare.
Thanks salius! I agree with what you said. In addition,
A lot of the value was just time set aside to thinking about management, so hard to separate that out without a control group
(but realistically, without the training, I would not have spent ~16 (including some additional work/loose threads after the workshop) hours thinking about management in one week.
So that alone is really valuable!
I feel like the implicit prioritization for some of the topics they covered possibly made more sense for experienced managers than people like me.
For example, about 1/3 of the time in the workshop was devoted to diversity/inclusion topics, and I'd be very surprised if optimal resource allocation for a new manager is anywhere close to 1/3 time spent on diversity/inclusion.
A really important point hammered throughout the training is the importance of clear and upfront communication.
Again, I think this is something I could have figured out through my usual combination of learning about things (introspection, asking around, internet research), but having this hammered in me is actually quite valuable.
I find a lot of the specific tools they suggested intuitively useful (eg, explicit MOCHA diagrams) , but I think I have to put in work to use them in my own management (and indeed failed to do this my first week as a manager).
I think high probability of existential risk this century mostly dampens the force of fanaticism*-related worries/arguments. I think this is close to self-evident, though can expand on it if it's interesting.
*fanaticism in the sense of very small probabilities of astronomically high payoffs, not in the everyday sense of people might do extreme actions because they're ideological. The latter is still a worry.
I find the unilateralist’s curse a particularly valuable concept to think about. However, I now worry that “unilateralist” is an easy label to tack on, and whether a particular action is unilateralist or not is susceptible to small changes in framing.
Consider the following hypothetical situations:
Company policy vs. team discretion
Alice is a researcher in a team of scientists at a large biomedical company. While working on the development of an HIV vaccine, the team accidentally created an air-transmissible variant of HIV. The scientists must decide whether to publish their discovery with the rest of the company, knowing that leaks may exist, and the knowledge may be used to create a devastating biological weapon, but also that it could help those who hope to develop defenses against such weapons, including other teams within the same company. Most of the team thinks they should keep it quiet, but company policy is strict that such information must be shared with the rest of the company to maintain the culture of open collaboration.
Alice thinks the rest of the team should either share this information or quit. Eventually, she tells her vice president her concer
I really like this (I think you could make it top level if you wanted). I think
these of these are cases of multiple levels of cooperation. If you're part of an
organization that wants to be uncooperative (and you can't leave cooperatively),
then you're going to be uncooperative with one of them.
3
Linch
3y
Good point. Now that you bring this up, I vaguely remember
[https://www.reddit.com/r/IAmA/comments/1aje16/we_are_david_sloan_wilson_evolutionary_biologist/c8xyn5x/]
a Reddit AMA where an evolutionary biologist made the (obvious in hindsight, but
never occurred to me at the time) claim that with multilevel selection, altruism
on one level is often means defecting on a higher (or lower) level. Which
probably unconsciously inspired this post!
As for making it top level, I originally wanted to include a bunch of thoughts
on the unilateralist's curse as a post, but then I realized that I'm a one-trick
pony in this domain...hard to think of novel/useful things that Bostrom et. al
hasn't already covered!
I'm worried about a potential future dynamic where an emphasis on forecasting/quantification in EA (especially if it has significant social or career implications) will have adverse effects on making people bias towards silence/vagueness in areas where they don't feel ready to commit to a probability forecast.
I think it's good that we appear to be moving in the direction of greater quantification and being accountable for probability estimates, but I think there's the very real risk that people see this and then become scared of committing their loose thoughts/intuitive probability estimates on record. This may result in us getting overall worse group epistemics because people hedge too much and are unwilling to commit to public probabilities.
See analogy to Jeff Kaufman's arguments on responsible transparency consumption:
I think many individual EAs should spend some time brainstorming and considering ways they can be really ambitious, eg come up with concrete plans to generate >100M in moral value, reduce existential risk by more than a basis point, etc.
Likewise, I think we as a community should figure out better ways to help people ideate and incubate such projects and ambitious career directions, as well as aim to become a community that can really help people both celebrate successes and to mitigate the individual costs/risks of having very ambitious plans fail.
I'm now pretty confused about whether normative claims can be used as evidence in empirical disputes. I generally believed no, with the caveat that for humans, moral beliefs are built on a scaffolding of facts, and sometimes it's easier to respond to an absurd empirical claim with the moral claim that has the gestalt sense of empirical beliefs if there isn't an immediately accessible empirical claim.
I talked to a philosopher who disagreed, and roughly believed that strong normative claims can be used as evidence against more confused/less c... (read more)
I haven't really thought about it, but it seems to me that if an empirical claim
implies an implausible normative claim, that lowers my subjective probability of
the empirical claim.
I actually found this article surprisingly useful/uplifting, and I suspect reorienting my feelings towards this approach is both more impactful and more emotionally healthy for me. I think recently (especially in the last month), I was getting pretty whiny/upset about the ways in which the world feels unfair to me, in mostly not-helpful ways.
I know that a lot of the apparent unfairness is due to personal choices that I on reflection endorse (though am not necessarily in-the-moment happy about). However, I suspect there's something true and important about ... (read more)
Re the recent discussions on whether EAs are overspending on luxuries for ourselves, one thing that strikes me is that EA is an unusually international and online movement. This means that many of us will come from very different starting contexts in terms of national wealth, and/or differing incomes or social class. So a lot of the clashes in conceptions of what types of spending seems "normal" vs "excessive" will come from pretty different priors/intuitions of what seems "normal." For example, whether it's "normal" to have >100k salaries, whether it's "normal" to have conferences in fancy hotels, whether it's normal to have catered food, etc.
There are various different framings here. One framing that I like is that (for some large subset of funded projects) the EA elites often think that your work is more than worth the money. So, it's often worth recalibrating your expectations of what types of spending is "normal," and make adequate time-money tradeoffs accordingly. This is especially the case if you are seen as unusually competent, but come from a country or cultural background that is substantially poorer or more frugal than the elite US average.
Another framing that's worthwhile is to make explicit time-money tradeoffs and calculations more often.
I could see this in myself: my parents were fairly middle-of-the-road first gen
Chinese American immigrants on scholarships, so I grew up in substantially
poorer environments than I think most EAs are used to (but by no means poor
compared to the rest of the world). So from that lens (and indeed, the lens of
that of many of my classmates from colleges in the American Midwest who have
trouble finding high-paying jobs), many of the salaries and perks at EA jobs,
conferences, etc., are quite excessive.
But after joining EA, I worked in Silicon Valley/tech for a while*, and now my
reference class for salaries/perks is probably more in line with that of the
Western elite. And in that regard EA conferences and perks seems slightly
fancier but not unduly so, and certainly the financial compensation is far
lower.
*As a sidenote, at the time the prevailing meme in at least some parts of EA was
"if you can't get a better job elsewhere, just work in FAANG and earn-to-give
while figuring out what to do next." I think that meme is less popular now. I
think I still recommend it for people in a position to do so, except for the
(significant) danger that working in FAANG (or comparable positions) can end up
making you too "soft" and unwilling to e.g. take risks with your career. A
related worry is that people in such positions can easily price themselves out
of [https://forum.effectivealtruism.org/s/9g2DikiZmbrTJTRRj/p/KbmHk7sJdRtmCgM2m]
doing ambitious altruistic efforts.
Very instructive anecdote on motivated reasoning in research (in cost-effectiveness analyses, even!):
Back in the 90’s I did some consulting work for a startup that was developing a new medical device. They were honest people–they never pressured me. My contract stipulated that I did not have to submit my publications to them for prior review. But they paid me handsomely, wined and dined me, and gave me travel opportunities to nice places. About a decade after that relationship came to an end, amicably, I had occasion to review the article I had published about the work I did for them. It was a cost-effectiveness analysis. Cost-effectiveness analyses have highly ramified gardens of forking paths that biomedical and clinical researchers cannot even begin to imagine. I saw that at virtually every decision point in designing the study and in estimating parameters, I had shaded things in favor of the device. Not by a large amount in any case, but slightly at almost every opportunity. The result was that my “base case analysis” was, in reality, something more like a “best case” analysis. Peer review did not discover any of this during the publication process, because each individual esti
In the Precipice, Toby Ord very roughly estimates that the risk of extinction from supervolcanoes this century is 1/10,000 (as opposed to 1/10,000 from natural pandemics, 1/1,000 from nuclear war, 1/30 from engineered pandemics and 1/10 from AGI). Should more longtermist resources be put into measuring and averting the worst consequences of supervolcanic eruption?
More concretely, I know a PhD geologist who's interested in doing an EA/longtermist career and is currently thinking of re-skilling for AI policy. Given that (AFAICT) literally zero people in our community currently works on supervolcanoes, should I instead convince him to investigate supervolcanoes at least for a few weeks/months?
If he hasn't seriously considered working on supervolcanoes before, then it definitely seems worth raising the idea with him.
I know almost nothing about supervolcanoes, but, assuming Toby's estimate is reasonable, I wouldn't be too surprised if going from zero to one longtermist researcher in this area is more valuable than adding an additional AI policy researcher.
There should maybe be an introductory guide for new LessWrong users coming in from the EA Forum, and vice versa.
I feel like my writing style (designed for EAF) is almost the same as that of LW-style rationalists, but not quite identical, and this is enough to be substantially less useful for the average audience member there.
For example, this identical question is a lot less popular on LessWrong than on the EA Forum, despite naively appearing to appeal to both audiences (and indeed if I were to guess at the purview of LW, to be cl... (read more)
I do agree that there are notable differences in what writing styles are often
used and appreciated on the two sites.
Could this also be simply because of a difference in the extent to which people
already know your username and expect to find posts from it interesting on the
two sites? Or, relatedly, a difference in how many active users on each site you
know personally?
I'm not sure how much those factors affect karma and comment numbers on either
site, but it seems plausible that the have a substantial affect (especially
given how an early karma/comment boost can set off a positive feedback loop).
Also, have you crossposted many things and noticed this pattern, or was it just
a handful? I think there's a lot of "randomness" in karma and comment numbers on
both sites, so if it's just been a couple crossposts it seems hard to be
confident that any patterns would hold in future.
Personally, when I've crossposted something to the EA Forum and to LessWrong,
those posts have decently often gotten more karma on the Forum and decently
often the opposite, and (from memory) I don't think there's been a strong
tendency in one direction or the other.
2
Linch
2y
Yeah I think this is plausible. Pretty unfortunate though.
I don't ever recall having a higher karma on LW than the Forum, though I
wouldn't be surprised if it happened once or twice.
One thing I'd be excited to see/fund is a comprehensive survey/review of what being an independent researcher in EA is like, and what are the key downsides, both:
a. From a personal perspective. What's unpleasant, etc about the work.
b. From a productivity perspective. What are people missing in independent research that's a large hit to their productivity and/or expected impact in the world.
This is helpful for two reasons:
More information/communication is generally helpful. I think having a clearer-eyed understanding of the downsides of independent research
Something that came up with a discussion with a coworker recently is that often internet writers want some (thoughtful) comments, but not too many, since too many comments can be overwhelming. Or at the very least, the marginal value of additional comments is usually lower for authors when there are more comments.
However, the incentives for commentators is very different: by default people want to comment on the most exciting/cool/wrong thing, so internet posts can easily by default either attract many comments or none. (I think) very little self-policing is done, if anything a post with many comments make it more attractive to generate secondary or tertiary comments, rather than less.
Meanwhile, internet writers who do great work often do not get the desired feedback. As evidence: For ~ a month, I was the only person who commented on What Helped the Voiceless? Historical Case Studies (which later won the EA Forum Prize).
This will be less of a problem if internet communication is primarily about idle speculations and cat pictures. But of course this is not the primary way I and many others on the Forum engage with the internet. Frequently, the primary publication v... (read more)
I think these are useful observations and questions. (Though I think "too many
comments" should probably be much less of a worry than "too few", at least if
the comments make some effort to be polite and relevant, and except inasmuch as
loads of comments on one thing sucks up time that could be spent commenting on
other things where that'd be more useful.)
I think a few simple steps that could be taken by writers are:
1. People could more often send google doc drafts to a handful of people
specifically selected for being more likely than average to (a) be
interested in reading the draft and (b) have useful things to say about it
2. People could more often share google doc drafts in the Effective Altruism
Editing & Review Facebook group
3. People could more often share google doc drafts in other Facebook groups,
Slack workspaces, or the like
* E.g., sharing a draft relevant to improving institutional decision-making
in the corresponding Facebook group
4. People could more often make posts/shortforms that include an executive
summary (or similar) and a link to the full google doc draft, saying that
this is still like a draft and they'd appreciate comment
* Roughly this has been done recently by Joe Carlsmith and Ben Garfinkel,
for example
* This could encourage more comments that just posting the whole thing to
the Forum as a regular post, since (a) this conveys that this is still a
work-in-progress and that comments are welcome, and (b) google docs make
it easier to comment on specific points
5. When people do post full versions of things on the Forum (or wherever), they
could explicitly indicate that they're interested in feedback, indicate
roughly what kinds of feedback would be most valuable, and indicate that
they might update the post in light of feedback (if that's true)
6. People could implement the advice given in these two good posts:
1. https://forum.effectivealtruism
2
MichaelA
2y
One other semi-relevant thing from my post Notes on EA-related research,
writing, testing fit, learning, and the Forum
[https://forum.effectivealtruism.org/posts/J7PsetipHFoj2Mv7R/notes-on-ea-related-research-writing-testing-fit-learning]:
One obvious reason against this is that maybe EA is too broad and the numbers we actually care about are too domain specific to specific queries/interests, but nonetheless I still think it's worth investigating.
I love this idea! Lots of fun ways to make infographics out of this, too.
Want to start out by turning this into a Forum question where people can suggest
numbers they think are important? (If you don't, I plan to steal your idea for
my own karmic benefit.)
2
Linch
2y
Thanks for the karmically beneficial tip!
I've now posted this question
[https://forum.effectivealtruism.org/posts/ekWRyJr9JneoWe5eH/what-are-some-key-numbers-that-almost-every-ea-should-know]
in its own right.
I think I have a preference for typing "xrisk" over "x-risk," as it is easier to type out, communicates the same information, and like other transitions (e-mail to email, long-termism to longtermism), the time has come for the unhyphenated version.
I think it's really easy to get into heated philosophical discussions about whether EAs overall use too much or too little jargon. Rather than try to answer this broadly for EA as a whole, it might be helpful for individuals to conduct a few quick polls to decide for themselves whether they ought to change their lexicon.
I'm curious if any of the researchers/research managers/academics/etc here a) read High Output Management b) basically believe in High Output Management's core ideas and c) have thoughts on how should it be adapted to research.
The core idea there is the production process:
"Deliver an acceptable product at the scheduled time, at the lowest possible cost."
The arguments there seem plausible and Andy Grover showed ingenuity in adapting the general idea into detailed stories for very different processes (eg running a restaraunt, training sales... (read more)
I've finally read the Huw Hughes review of the CE Delft Techno-Economic Analyses (our summary here) of cultured meat and thought it was interesting commentary on the CE Delft analysis, though less informative on the overall question of cultured meat scaleups than I hoped.
Overall their position on CE Delft's analysis was similar to ours, except maybe more bluntly worded. They were more critical in some parts and less critical in others.
Things I liked about the Hughes review:
the monoclonal antibodies reference class point was interesting and not something I've considered before
I liked how it spelled out the differences in stability between conventional meat and cell slurry; I've thought about it before but didn't seriously consider it, having this presented so clearly was useful
I liked the author diagramming an entire concept area to see what a real factory might look like, instead of only looking at the myocyte cell process
I fully agree with the author about the CE Delft TEA being way too underspecified, as well as the aseptic conditions stuff making food bioreactors a very unlikely reference class (though of course I already knew these points)
For those interested, here are the paragraphs that added new information on the
CE Delft TEA that I hadn't considered or seen in other TEAs or reviews.
Cell growth
"For bacteria, one may generally subculture in the range of 1:10 to 1:100,
depending on the bacterium. For stem cells, which are more fastidious, the
highest subculture ratio (i.e. the amount of an old culture one uses to seed or
start a new culture) is typically 1:5. This is significantly less than the 1:200
cell ratio that is proposed in the TEA. Experience dictates that if stem cells
are sub-cultured at the ratio proposed, they will either differentiate and stop
growing, or simply will die. "
Medium costs
"In an analysis of culture medium costs, Specht (2019) used DMEM/F12 medium as a
base, with a cost of $62,400 for a 20,000 L batch (or $3.12/L). However, this
appears to be the cost for powdered bulk medium (from, e.g., MP Bio), and does
not include the cost for labour, USP water, formulation, filtration and testing
prior to culture. Given the fact that up to now stem cells require custom media,
the price for base medium alone could rise to $822,000 for the same sized
(20,000 L)batch. However, it should be noted that a properly developed medium
may rely less on growth factor additives" This is also used by Risner, et al
(2020) in their "this is where the miracle happens" Scenario 4.
"insulin costs will likely remain as they are due to current and future
manufacturing standards and volumes. "
Contamination
"The building will require air locks with increasing air handling quality, for
both materials and personnel. Typically this comprises a Class D corridor, with
Class C rooms except where open phase is carried out, which must be Class A in a
Class B environment with Class C minimal changing rooms. The TEA document does
not take into account this quality standard, nor does it take into account the
additional personnel time"
Stability
“Cell pastes or slurries are notoriously unstable and will de
On the forum, it appears to have gotten harder for me to do multiple quote blocks in the same comment. I now often have to edit a post multiple times so quoted sentences are correctly in quote blocks, and unquoted sections are not. Whereas in the past I do not recall having this problem?
I'm going to guess that the new editor is the difference between now and
previously. What's the issue you're seeing? Is there a difference between the
previewed and rendered text? Ideally you could get this to repro on LessWrong's
development server [https://baserates.org/], which would be useful for bug
reports, but no worries if not.
Sometimes I hear people who caution humility say something like "this question has stumped the best philosophers for centuries/millennia. How could you possibly hope to make any progress on it?". While I concur that humility is frequently warranted and that in many specific cases that injunction is reasonable [1], I think the framing is broadly wrong.
In particular, using geologic time rather than anthropological time hides the fact that there probably weren't that many people actively thinking about these issues, especially carefully, in a sustained way, and making sure to build on the work of the past. For background, 7% of all humans who have ever lived are alive today, and living people compose 15% of total human experience [2] so far!!!
It will not surprise me if there are about as many living philosophers today as there were dead philosophers in all of written history.
For some specific questions that particularly interest me (eg. population ethics, moral uncertainty), the t
In Twitter and elsewhere, I've seen a bunch of people argue that AI company execs and academics are only talking about AI existential risk because they want to manufacture concern to increase investments and/or as a distraction away from near-term risks and/or regulatory capture. This is obviously false.
However, there is a nearby argument that is likely true: which is that incentives drive how people talk about AI risk, as well as which specific regulations or interventions they ask for. This is likely to happen both explicitly and unconsciously. It's important (as always) to have extremely solid epistemics, and understand that even apparent allies may have (large) degrees of self-interest and motivated reasoning.
Safety-washing is a significant concern; similar things have happened a bunch in other fields, it likely has already happened a bunch in AI, and will likely happen again in the months and years to come, especially if/as policymakers and/or the general public become increasingly uneasy about AI.
Contra claims like here and here, I think extraordinary evidence is rare for probabilities that are quasi-rational and mostly unbiased, and should be quite shocking when you see it. I'd be interested in writing an argument for why you should be somewhat surprised to see what I consider extraordinary evidence[1]. However, I don't think I understand the "for" case for extraordinary evidence being common[2], so I don't understand the case for it and can't present the best "against" case.
[1] operationalized e.g. as a 1000x or 10000x odds update on a question t... (read more)
tl;dr:
In the context of interpersonal harm:
1. I think we should be more willing than we currently are to ban or softban people.
2. I think we should not assume that CEA's Community Health team "has everything covered"
3. I think more people should feel empowered to tell CEA CH about their concerns, even (especially?) if other people appear to not pay attention or do not think it's a major concern.
4. I think the community is responsible for helping the CEA CH team with having a stronger mandate to deal with interpersonal harm, including some degree of acceptance of mistakes of overzealous moderation.
(all views my own) I want to publicly register what I've said privately for a while:
For people (usually but not always men) who we have considerable suspicion that they've been responsible for significant direct harm within the community, we should be significantly more willing than we currently are to take on more actions and the associated tradeoffs of limiting their ability to cause more harm in the community.
Some of these actions may look pretty informal/unofficial (gossip, explicitly warning newcomers against specific people, keep an unofficial eye out for some people during par... (read more)
Thank you so much for laying out this view. I completely agree, including every single subpoint (except the ones about the male perspective which I don't have much of an opinion on). CEA has a pretty high bar for banning people. I'm in favour of lowering this bar as well as communicating more clearly that the bar is really high and therefore someone being part of the community certainly isn't evidence they are safe.
Thank you in particular for point D. I've never been quite sure how to express the same point and I haven't seen it written up elsewhere.
It's a bit unfortunate that we don't seem to have agreevote on shortforms.
If your civilization's discount rate is too low, you'll mug yourself trying to prevent the heat death of the universe. If your discount rate is too high, you'll mug yourself wasting resources attempting time travel. The correct answer lies somewhere in between.
I think a subtext for some of the EA Forum discussions (particularly the more controversial/ideological ones) is that a) often two ideological camps form, b) many people in both camps are scared, c) ideology feeds on fear and d) people often don't realize they're afraid and cover it up in high-minded ideals (like "Justice" or "Truth").
I think if you think other EAs are obviously, clearly Wrong or Evil, it's probably helpful to
a) realize that your interlocutors (fellow EAs!) are human, and most of them are here because they want to serve the good
b) internally try to simulate their object-level arguments
c) try to understand the emotional anxieties that might have generated such arguments
d) internally check in on what fears you might have, as well as whether (from the outside, looking from 10,000 feet up) you might acting out the predictable moves of a particular Ideology.
e) take a deep breath and a step back, and think about your intentions for communicating.
A Personal Apology
I think I’m significantly more involved than most people I know in tying the fate of effective altruism in general, and Rethink Priorities in particular, with that of FTX. This probably led to rather bad consequences ex post, and I’m very sorry for this.
I don’t think I’m meaningfully responsible for the biggest potential issue with FTX. I was not aware of the alleged highly unethical behavior (including severe mismanagement of consumer funds) at FTX. I also have not, to my knowledge, contributed meaningfully to the relevant reputational laundering or branding that led innocent external depositors to put money in FTX. The lack of influence there is not because I had any relevant special knowledge of FTX, but because I have primarily focused on building an audience within the effective altruism community, who are typically skeptical of cryptocurrency, and because I have actively avoided encouraging others to invest in cryptocurrency. I’m personally pretty skeptical about the alleged social value of pure financialization in general and cryptocurrency in particular, and also I’ve always thought of crypto as a substantially more risky asset than many retail invest... (read more)
In replies to this thread, here are some thoughts I have around much of the discourse that have come out so far about recent controversies. By "discourse," I'm thinking of stuff I mostly see here, on EA Twitter, and EA Facebook. I will not opine on the details of the controversies themselves. Instead I have some thoughts on why I think the ensuing discourse is mostly quite bad, some attempts to reach an understanding, thoughts on how we can do better, as well as some hopefully relevant tangents.
I split my thoughts into multiple comments so people can upvote or downvote specific threads.
While I have thought about this question a bunch, these comments has been really hard for me to write and the final product is likely pretty far from what I’d like, so please bear with me. As usual, all errors are my own.
We (EA Forum) are maybe not strong enough (yet?) to talk about certain topics
A famous saying in LessWrong-speak is "Politics is the Mind-Killer". In context, the post was about attempting to avoid using political examples in non-political contexts, to avoid causing people to become less rational with political blinders on, and also to avoid making people with different politics feel unwelcome. More broadly, it's been taken by the community to mean a general injunction against talking about politics when unnecessary most of the time.
Likewise, I think there are topics that are as or substantially more triggering of ideological or tribal conflict as modern partisan politics. I do not think we are currently strong enough epistemically, or safe enough emotionally, to be able to discuss those topics with the appropriate level of nuance and intellect and tact. Except for the topics that are extremely decision-relevant (e.g. "which UK political party should I join to reduce malaria/factory farming/AI doom probabilities") I will personally prefer that we steer clear of them for now, and wait until our epistemics and cohesion are one day perhaps good enough to approach them.
Some counters to grandiosity
Some of my other comments have quite grandiose language and claims. In some ways this is warranted: the problems we face are quite hard. But in other ways perhaps the grandiosity is a stretch: we have had a recent surge of scandals, and we'll like have more scandals in the years and perhaps decades to come. We do need to be somewhat strong to face them well. But as Ozzie Gooen rightfully point out, in contrast to our historical moral heroes[1], the problems we face are pretty minor in comparison.
In comparison, the problems of our movement just seems kind of small in comparison? "We kind of went down from two billionaires and very little political/social pushback, to one billionaire and very little political/social pushback?" A few people very close to us committed crimes? We had one of our intellectual heavyweights say something very racist 20+ years ago, and then apologized poorly? In the grand arc of ... (read more)
A plea for basic kindness and charity
I think many people on both sides of the discussion
Perhaps I’m just reasserting basic forum norms, but I think we should instead at least try to interpret other people on this forum more charitably. Moreover, I think we should generally try to be kind to our fellow EAs[1]. Most of us are here to do good. Many of us have made substantial sacrifices in order to do so. We may have some disagreements and misunderstandings now, and we likely will again in the future, but mora... (read more)
Some gratitude for the existing community
It’s easy to get jaded about this, but in many ways I find the EA community genuinely inspirational. I’m sometimes reminded of this when I hear about a new good thing EAs have done, and at EA Global, and when new EAs from far-away countries reach out to me with a specific research or career question. At heart, the EA community is a community of thousands of people, many of whom are here because they genuinely want to do the most good, impartially construed, and are actively willing to use reason and evidence to get there. This is important, and rare, and I think too easily forgotten.
I think it's helpful to think about a few things you're grateful for in the community (and perhaps even your specific interlocutors) before engaging in heated discourse.
Your forum contributions in recent months and this thread in particular 🙏🙏🙏
Talk to people, not at people
In recent days, I've noticed an upsurge of talking at people rather than with them. I think there's something lost here, where people stopped assuming interlocutors are (possibly mistaken) fellow collaborators in the pursuit of doing good, but more like opponents to be shot down and minimized. I think something important is lost both socially and epistemically when we do this, and it's worthwhile to consider ways to adapt a more collaborative mindset. Some ideas:
1. Try to picture yourself in the other person's shoes. Try to understand, appreciate, and anticipate both their worries and their emotions before dashing off a comment.
2. Don't say "do better, please" to people you will not want to hear the same words from. It likely comes across as rather patronizing, and I doubt the background rates of people updating positively from statements like that is particularly high.
3. In general, start with the assumption of some basic symmetry on how and what types of feedback you'd like to receive before providing it to others.
Enemy action?
I suspect at least some of the optics and epistemics around the recent controversies are somewhat manipulated by what I call "enemy action." That is, I think there are people out there who are not invested in this project of doing good, and are instead, for idiosyncratic reasons I don't fully understand[1], interested in taking our movement down. This distorts a) much of the optics around the recent controversies, b) much of the epistemics in what we talk about and what we choose to pay attention to and c) much of our internal sense of cohesion.
I don't have strong evidence of this, but I think it is plausible that at least some of the current voting on the forum on controversial issues is being manipulated by external actors in voting rings. I also think it is probable that some quotes from both on and off this forum are selectively mined in external sources, so if you come to the controversies from them, you should make take a step back and think of ways in which your epistemics or general sense of reality is being highjacked. Potential next steps:
- Keep your cool
- Assume good faith from community members most of the time
- If someone has a known history of repeatedly
... (read more)We need to become stronger
I'm not sure this comment is decision-relevant, but I want us to consider the need for us, both individually and collectively, to become stronger. We face great problems ahead of us, and we may not be able up for the challenge. We need to face them with intellect, and care, and creativity and reason. We need to face them with cooperation, and cohesion, and love for fellow man, but also strong independence and skepticism and ability to call each out on BS.
We need to be clear enough in our thinking to identify the injustices in the world, careful enough in our planning to identify the best ways to fight them, and committed and steady enough in our actions to decisively act when we need to. We need to approach the world with fire in our hearts and ice in our veins.
We should try to help each other identify, grow, and embody the relevant abilities and virtues needed to solve the world's most pressing problems. We should try our best to help each other grow together.
This may not be enough, but we should at least try our best.
Morality is hard, and we’re in this together.
One basic lesson I learned from trying to do effective altruism for much of my adult life is that morality is hard. Morality is hard at all levels of abstraction: Cause prioritization, or trying to figure out the most pressing problems to work on, is hard. Intervention prioritization, or trying to figure out how we can tackle the most important problems to work on, is hard. Career choice, or trying to figure out what I personally should do to work on the most important interventions for the most important problems is hard. Day-to-day prioritization is hard. In practice, juggling a long and ill-defined list of desiderata to pick the morally least-bad outcome is hard. And dedication and commitment to continuously hammer away at doing the right thing is hard.
And the actual problems we face are really hard. Millions of children die every year from preventable causes. Hundreds of billions of animals are tortured in factory farms. Many of us believe that there are double-digit percentage points of existential risk this century. And if we can navigate all the perils and tribulations of this century, we still need to prepare our descendant... (read more)
Anthropic awareness or “you’re not just in traffic, you are traffic.”
An old standup comedy bit I like is "You're not in traffic, you are traffic."Traffic isn't just something that happens to you, but something you actively participate in (for example, by choosing to leave work during rush hour). Put another way, you are other people's traffic.
I take the generalized version of this point pretty seriously. Another example of this was I remember complaining about noise at a party. Soon after, I realized that the noise I was complaining about was just other people talking! And of course I participated in (and was complicit in) this issue.
Similarly, in recent months I complained to friends about the dropping kindness and epistemic standards on this forum. It took me way too long to realize the problem with that statement, but the reality is that discourse, like traffic, isn't something that just happens to me. If anything, as one of the most active users on this forum, I'm partially responsible for the dropping forum standards, especially if I don't active try to make epistemic standards better.
So this thread is my partial attempt to rectify the situation.
I'd love ... (read more)
Understanding and acknowledging the subtext of fear
I think a subtext for some of the EA Forum discussions (particularly the more controversial/ideological ones) is that a) often two ideological camps form, b) many people in both camps are scared, c) ideology feeds on fear and d) people often don't realize they're afraid and cover it up in high-minded ideals (like "Justice" or "Truth")[1].
I think if you think other EAs are obviously, clearly Wrong or Evil, it's probably helpful to
a) realize that your interlocutors (fellow EAs!) are human, and most of them are here because they want to serve the good
b) internally try to simulate their object-level arguments
c) try to understand the emotional anxieties that might have generated such arguments
d) internally check in on what fears you might have, as well as whether (from the outside, looking from 10,000 feet up) you might acting out the predictable moves of a particular Ideology.
e) take a deep breath and a step back, and think about your intentions for communicating.
- ^
... (read more)In the draft of a low-winded post I probably will never publish, I framed it thusly: "High contextualizers are scared. (They may not reali
Bystanders exist
When embroiled in ideological conflict, I think it's far too easy to be ignorant of (or in some cases, deliberately downplay for bravado reasons) the existence of bystanders to your ideological war. For example, I think some black EAs are directly hurt by the lack of social sensitivity displayed in much of the discourse around the Bostrom controversy (and perhaps the discussions themselves). Similarly, some neurodivergent people are hurt by the implication that maximally sensitive language is a desiderata on the forum, and the related implication that people like them are not welcome. Controversies can also create headaches for community builders (including far away from the original controversy), for employees at the affected or affiliated organizations, and for communications people more broadly.
The move to be making is to stop for a bit. Note that people hurting are real people, not props. And real people could be seriously hurting for reasons other than direct ideological disagreement.
While I think it is tempting to use bystanders to make your rhetoric stronger, embroiling bystanders in your conflict is I think predictably bad. If you know people who you think m... (read more)
The whole/only real point of the effective altruism community is to do the most good.
If the continued existence of the community does the most good,
I desire to believe that the continued existence of the community does the most good;
If ending the community does the most good,
I desire to believe that ending the community does the most good;
Let me not become attached to beliefs I may not want.
Red teaming papers as an EA training exercise?
I think a plausibly good training exercise for EAs wanting to be better at empirical/conceptual research is to deep dive into seminal papers/blog posts and attempt to identify all the empirical and conceptual errors in past work, especially writings by either a) other respected EAs or b) other stuff that we otherwise think of as especially important.
I'm not sure how knowledgeable you have to be to do this well, but I suspect it's approachable for smart people who finish high school, and certainly by the time they finish undergrad^ with a decent science or social science degree.
I think this is good career building for various reasons:
- you can develop a healthy skepticism of the existing EA orthodoxy
- I mean skepticism that's grounded in specific beliefs about why things ought to be different, rather than just vague "weirdness heuristics" or feeling like the goals of EA conflict with other tribal goals.
- (I personally have not found high-level critiques of EA, and I have read many, to be particularly interesting or insightful, but this is just a personal take).
- you actually deeply understand at least one topic well enough
... (read more)This is another example of a Shortform that could be an excellent top-level post (especially as it's on-theme with the motivated reasoning post that was just published). I'd love to see see this spend a week on the front page and perhaps convince some readers to try doing some red-teaming for themselves. Would you consider creating a post?
Upon (brief) reflection I agree that relying on the epistemic savviness of the mentors might be too much and the best version of the training program will train a sort of keen internal sense of scientific skepticism that's not particularly reliant on social approval.
If we have enough time I would float a version of a course that slowly goes from very obvious crap (marketing tripe, bad graphs) into things that are subtler crap (Why We Sleep, Bem ESP stuff) into weasely/motivated stuff (Hickel? Pinker? Sunstein? popular nonfiction in general?) into things that are genuinely hard judgment calls (papers/blog posts/claims accepted by current elite EA consensus).
But maybe I'm just remaking the Calling Bullshit course but with a higher endpoint.
___
(I also think it's plausible/likely that my original program of just giving somebody an EA-approved paper + say 2 weeks to try their best to Red Team it will produce interesting results, even without all these training wheels).
One additional risk: if done poorly, harsh criticism of someone else's blog post from several years ago could be pretty unpleasant and make the EA community seem less friendly.
I'm actually super excited about this idea though - let's set some courtesy norms around contacting the author privately before red-teaming their paper and then get going!
General suspicion of the move away from expected-value calculations and cost-effectiveness analyses.
This is a portion taken from a (forthcoming) post about some potential biases and mistakes in effective altruism that I've analyzed via looking at cost-effectiveness analysis. Here, I argue that the general move (at least outside of human and animal neartermism) away from Fermi estimates, expected values, and other calculations just makes those biases harder to see, rather than fix the original biases.
I may delete this section from the actual post as this point might be a distraction from the overall point.
____
I’m sure there are very good reasons (some stated, some unstated) for moving away from cost-effectiveness analysis. But I’m overall pretty suspicious of the general move, for a similar reason that I’d be suspicious of non-EAs telling me that we shouldn’t use cost-effectiveness analyses to judge their work, in favor of say systematic approaches, good intuitions, and specific contexts like lived experiences (cf. Beware Isolated Demands for Rigor):
... (read more)I think it would be valuable to see quantitative estimates of more problem areas and interventions. My order of magnitude estimate would be that if one is considering spending $10,000-$100,000, one should do a simple scale, neglectedness, and tractability analysis. But if one is considering spending $100,000-$1 million, one should do an actual cost-effectiveness analysis. So candidates here would be wild animal welfare, approval voting, improving institutional decision-making, climate change from an existential risk perspective, biodiversity from an existential risk perspective, governance of outer space etc. Though it is a significant amount of work to get a cost-effectiveness analysis up to peer review publishable quality (which we have found requires moving beyond Guesstimate, e.g. here and here), I still think that there is value in doing a rougher Guesstimate model and having a discussion about parameters. One could even add to one of our Guesstimate models, allowing a direct comparison with AGI safety and resilient foods or interventions for loss of electricity/industry from a long-term perspective.
New Project/Org Idea: JEPSEN for EA research or EA org Impact Assessments
Note: This is an updated version of something I wrote for “Submit grant suggestions to EA Funds”
What is your grant suggestion?
An org or team of people dedicated to Red Teaming EA research. Can include checks for both factual errors and conceptual ones. Like JEPSEN but for research from/within EA orgs. Maybe start with one trusted person and then expand outwards.
After demonstrating impact/accuracy for say 6 months, can become a "security" consultancy for either a) EA orgs interested in testing the validity of their own research or b) an external impact consultancy for the EA community/EA donors interested in testing or even doing the impact assessments of specific EA orgs. For a), I imagine Rethink Priorities may want to become a customer (speaking for myself, not the org).
Potentially good starting places:
- Carefully comb every chapter of The Precipice
- Go through ML/AI Safety papers and (after filtering on something like prestige or citation count) pick some papers at random to Red Team
- All of Tetlock's research on forecasting, particularly the ones with factoids most frequently cited in EA circle... (read more)
Honestly I don't understand the mentality of being skeptical of lots of spending on EA outreach. Didn't we have the fight about overhead ratios, fundraising costs, etc with Charity Navigator many years ago? (and afaict decisively won).
One thing I dislike about certain thought-experiments (and empirical experiments!) is that they do not cleanly differentiate between actions that are best done in "player vs player" and "player vs environment" domains.
For example, a lot of the force of our intuitions behind Pascal's mugging comes from wanting to avoid being "mugged" (ie, predictably lose resources to an adversarial and probably-lying entity). However, most people frame it as a question about small probabilities and large payoffs, without the adversarial component.
Similarly, empirical social psych experiments on hyperbolic discounting feel suspicious to me. Indifference between receiving $15 immediately vs $30 in a week (but less aggressive differences between 30 weeks and 31 weeks) might track a real difference in discount rates across time, or it could be people's System 1 being naturally suspicious that the experimenters would actually pay up a week from now (as opposed to immediately).
So generally I think people should be careful in thinking about, and potentially cleanly differentiating, the "best policy for making decisions in normal contexts" vs "best policy for making decisions in contexts where someone is actively out to get you."
One thing that confuses me is that the people saying "EAs should be more frugal" and the people saying "EAs should be more hardworking" are usually not the same people. This is surprising to me, since I would have guessed that answers to considerations like "how much should we care about the demands of morality" and "how much should we trade off first order impact for inclusivity" and "how much we should police fellow travelers instead of have more of a live-and-let-live attitude about the community" should cluster pretty closely together.
A general policy I've adapted recently as I've gotten more explicit* power/authority than I'm used to is to generally "operate with slightly to moderately more integrity than I project explicit reasoning or cost-benefits analysis would suggest."
This is primarily for epistemics and community epistemics reasons, but secondarily for optics reasons.
I think this almost certainly does risk leaving value on the table, but on balance it is a better balance than potential alternatives:
- Just following explicit reasoning likely leads to systematic biases "shading" the upsides higher and the downsides lower, and I think this is an explicit epistemics bias that can and should be corrected for.
- there is also a slightly adversarial dynamics on the optics framing -- moves that seem like a normal/correct amount of integrity to me may adversarially be read as lower integrity to others.
- Projections/forecasts of reasoning (which is necessary because explicit reasoning is often too slow) may additionally be biased on top of the explicit reasoning (I have some pointers here)
- Always "behaving with maximal integrity" probably leaves too much value on the table, unless you define integrity in a pre
... (read more)Target audience: urgent longtermists, particularly junior researchers and others who a) would benefit from more conceptual clarity on core LT issues, and b) haven’t thought about them very deeply.
Note that this shortform assumes but does not make arguments about a) the case for longtermism or b) the case for urgent (vs patient) longtermism, or c) the case that the probability of avertable existential risk this century is fairly high. It probably assumes other assumptions that are commonly held in EA as well.
___
Thinking about protecting the future in terms of extinctions, dooms, and utopias
When I talk about plans to avert existential risk with junior longtermist researchers and others, I notice many people, myself included, being somewhat confused about what we actually mean when we talk in terms of “averting existential risk” or “protecting the future.” I notice 3 different clusters of definitions that people have intuitive slippage between, where it might help to be more concrete:
1. Extinction – all humans and our moral descendants dying
2. Doom - drastic and irrevocable curtailing of our potential (This is approximately the standard definition)
3. (Not) Utopia - (... (read more)
Doom should probably include s-risk (i.e. fates worse than extinction).
I'm pretty confused about the question of standards in EA. Specifically, how high should it be? How do we trade off extremely high evidential standards against quantity, either by asking people/ourselves to sacrifice quality for quantity or by scaling up the number of people doing work by accepting lower quality?
My current thinking:
1. There are clear, simple, robust-seeming arguments for why more quantity* is desirable, in far mode.
2. Deference to more senior EAs seems to point pretty heavily towards focusing on quality over quantity.
3. When I look at specific interventions/grant-making opportunities in near mode, I'm less convinced they are a good idea, and lean towards earlier high-quality work is necessary before scaling.
The conflict between the very different levels of considerations in #1 vs #2 and #3 makes me fairly confused about where the imbalance is, but still maybe worth considering further given just how huge a problem a potential imbalance could be (in either direction).
*Note that there was a bit of slippage in my phrasing, while at the frontiers there's a clear quantity vs average quality tradeoff at the output level, the function that translates inp... (read more)
Something that I think is useful to keep in mind as you probe your own position, whether by yourself, or in debate with others, is:
I think sometimes I e.g. have a object-level disagreement with someone about a technology or a meta-disagreement about the direction of EA strategy, and my interlocutor says something like "oh I'll only change my mind if you demonstrate that my entire understanding of causality is wrong or everything I learned in the... (read more)
The General Longtermism team at Rethink Priorities is interested in generating, fleshing out, prioritizing, and incubating longtermist megaprojects.
But what are longtermist megaprojects? In my mind, there are tentatively 4 criteria:
- Scale of outcomes: The outputs should be large in scope, from a longtermist point of view. A project at scale should have a decent shot of reducing probability of existential risk by a large amount. Perhaps we believe that after considerable research, we have a decent chance of concluding that the project will reduce existential risk by >0.01% (a "basis point").*
- Cost-effectiveness: The project should have reasonable cost-effectiveness. Given limited resources, we shouldn't spend all of them on a project that cost a lot of money and human capital for merely moderate gain. My guess is that we ought to have a numerical threshold of between 100M and 3B (best guess for current target: 500M) for financial plus human capital cost per basis point of existential risk averted.
- Scale of inputs: The inputs should be fairly large as well. An extremely impressive paper or a single conversation with the right person, no matter how impactful, should not count as
... (read more)Meta: It seems to me that the EA community talks about "red teaming" our own work a lot more often than they did half a year ago. It's unclear to me how much my own shortforms instigated this, vs. e.g. independent convergence.
This seems like a mildly important thing for me to track, as it seems important to me to gauge what fraction of my simple but original-ish ideas are only counterfactually a few months "ahead of the curve," vs genuinely novel and useful for longer.
To me, the core of what EAs gesture at when referring to human agency is represented by the following quotes:
1.
2.
3.
4.
I wrote some lines about what I see as the positive track record of utilitarianism in general and Bentham in particular.
Clarification on my own commenting norms:
If I explicitly disagreed with a subpoint in your post/comment, you should assume that I'm only disagreeing with that subpoint; you should NOT assume that I disagree with the rest of the comment and are only being polite. Similarly, if I reply with disagreement to a comment or post overall, you should NOT assume I disagree with your other comments or posts, and certainly I'm almost never trying to admonish you as a person. Conversely, agreements with subpoints should not be treated as agreements with your overall point, agreements with the overall point of an article should not be treated as an endorsement of your actions/your organization, and so forth.
I welcome both public and private feedback on my own comments and posts, especially points that note if I say untrue things. I try to only say true things, but we all mess up sometimes. I expect to mess up in this regard more often than most people, because I'm more public with my output than most people.
Would you be interested in writing a joint essay or perspectives on being Asian American, maybe taking any number of angles, that you can decide upon (e.g. inside liminal spaces of this identity, inside subcultures of tech or EA culture?).
Something tells me you have useful opinions on this topic, and that these are different to mine.
Adding background for context/calibration
This might help you decide whether you want to engage:
- I personally have had good experiences or maybe I am blind and unduly privileged. This is basically what I’ll write. I’m OK if my content is small or a minority of the writing.
- I personally don’t use the approaches today’s activists use. Instead, I see activism as instrumental to EA cause areas. (Another way of looking at this is that if we thought it was the right thing to do, we could promote the relevant activism/causes to EA causes.)
- Based on the two points above, I do not want privilege or attention as a result of the essay or related activity (but I am happy if others seek it).
- Something that needs attention are efforts to pool Asian’s with Caucasians, as to cut off Asian (or at least Han Chinese) status as a distinct group with valid opinions o
... (read more)This seems interesting but I don't currently think it'd trade off favorably against EA work time. Some very quickly typed thoughts re: personal experiences (apologies if it sounds whiny).
1. I've faced minor forms of explicit racism and microaggressions, but in aggregate they're pretty small, possibly smaller than the benefit of speaking a second language and cultural associations.
2. I expect my life would be noticeably better if I were a demographically twin who's Caucasian. But I'm not confident about this.
3. This is almost entirely due to various subtle forms of discrimination at a statistical level, rather than any forms of outright aggression or discrimination.
3a) e.g. I didn't get admitted to elite or even top 50 colleges back in 2011 when the competition was much less stiff than now. I had 1570/1600 SAT, 7 APs (iirc mostly but not all 5s), essays that won minor state-level awards, etc. To be clear, I don't think my profile was amazing or anything but statistically I think my odds would've been noticeably higher if I weren't Asian. OTOH I'm not confident that elite college attendance does much for someone controlling for competence (I think mostly the costs are soc... (read more)
Thanks for writing this, this is thoughtful, interesting and rich in content. I think others benefitted from reading this too.
Also, a reason I asked was that I was worried about the chance that I was ignorant about Asian-American experiences for idiosyncratic reasons. The information you provided was useful.
There is other content you mentioned that seems important (3c and 6). I will send a PM related to this. Maybe there are others reading who know you who also would like to listen to your experiences on these topics.
Re the post On Elitism in EA. Here is the longer version of my thoughts before I realized it could be condensed a lot:
I don't think I follow your model. You define elitism the following way:
In other words, the "elite" is defined as people who... (read more)
Updated version on https://docs.google.com/document/d/1BDm_fcxzmdwuGK4NQw0L3fzYLGGJH19ksUZrRloOzt8/edit?usp=sharing
Cute theoretical argument for #flattenthecurve at any point in the distribution
- What is #flattenthecurve?
- The primary theory behind #flattenthecurve is assuming that everybody who will get COVID-19 will eventually get it anyway...is there anything else you can do?
- It turns out it’s very valuable to
- Delay the spread so that a) the peak of the epidemic spread is lower (#flattenthecurve)
- Also to give public health professionals, healthcare sy
... (read more)Should there be a new EA book, written by somebody both trusted by the community and (less importantly) potentially externally respected/camera-friendly?
Kinda a shower thought based on the thinking around maybe Doing Good Better is a bit old right now for the intended use-case of conveying EA ideas to newcomers.
I think the 80,000 hours and EA handbooks were maybe trying to do this, but for whatever reason didn't get a lot of traction?
I suspect that the issue is something like not having a sufficiently strong "voice"/editorial line, and what you want for a book that's a)bestselling and b) does not sacrifice nuance too much is one final author + 1-3 RAs/ghostwriters.
What will a company/organization that has a really important secondary mandate to focus on general career development of employees actually look like? How would trainings be structured, what would growth trajectories look like, etc?
When I was at Google, I got the distinct impression that while "career development" and "growth" were common buzzwords, most of the actual programs on offer were more focused on employee satisfaction/retention than growth. (For example, I've essentially never gotten any feedback on my selection of training courses or books that I bought with company money, which at the time I thought was awesome flexibility, but in retrospect was not a great sign of caring about growth on the part of the company).
Edit: Upon a reread I should mention that there are other ways for employees to grow within the company, eg by having some degree of autonomy over what projects they want to work on.
I think there are theoretical reasons for employee career growth being underinvested by default. Namely, that the costs of career growth are borne approximately equally between the employer and the employee (obviously this varies from case to case), whil... (read more)
Edit: By figuring out ethics I mean both right and wrong in the abstract but also what the world empirically looks like so you know what is right and wrong in the particulars of a situation, with an emphasis on the latter.
I think a lot about ethics. Specifically, I think a lot about "how do I take the best action (morally), given the set of resources (including information) and constraints (including motivation) that I have." I understand that in philosophical terminology this is only a small subsection of applied ethics, and yet I spend a lot of time thinking about it.
One thing I learned from my involvement in EA for some years is that ethics is hard. Specifically, I think ethics is hard in the way that researching a difficult question or maintaining a complicated relationship or raising a child well is hard, rather than hard in the way that regularly going to the gym is hard.
When I first got introduced to EA, I believed almost the opposite (this article presents something close to my past views well): that the hardness of living ethically is a matter of execution and will, rather than that of constantly making tradeoffs in a difficult-to-navigate domain.
I still ... (read more)
I'm at a good resting point in my current projects, so I'd like to take some time off to decide on "ambitious* projects Linch should be doing next," whether at RP or elsewhere.
Excited to call with people who have pitches, or who just want to be a sounding board to geek out with me.
*My current filter on "ambition" is “only consider projects with a moral value >> that of adding 100M to Open Phil’s coffers assuming everything goes well.” I'm open to arguments that this is insufficiently ambitious, too ambitious, or carving up the problem at the wrong level of abstraction.
One alternative framing is thinking of outputs rather than intermediate goals, eg, "only consider projects that can reduce x-risk by >0.01% assuming everything goes well."
A corollary of background EA beliefs is that everything we do is incredibly important.
This is covered elsewhere in the forum, but I think an important corollary of many background EA + longtermist beliefs is that everything we do is (on an absolute scale) very important, rather than useless.
I know some EAs who are dispirited because they donate a few thousand dollars a year when other EAs are able to donate millions. So on a relative scale, this makes sense -- other people are able to achieve >1000x the impact through their donations as you do.
But the "correct" framing (I claim) would look at the absolute scale, and consider stuff like we are a) among the first 100 billion or so people and we hope there will one day be quadrillions b) (most) EAs are unusually well-placed within this already very privileged set and c) within that even smaller subset again, we try unusually hard to have a long term impact, so that also counts for something.
EA genuinely needs to prioritize very limited resources (including time and attention), and some of the messages that radiate from our community, particularly around relative impact of different people, may come across as... (read more)
I don't know, but my best guess is that "janitor at MIRI"-type examples reinforce a certain vibe people don't like — the notion that even "lower-status" jobs at certain orgs are in some way elevated compared to other jobs, and the implication (however unintended) that someone should be happy to drop some more fulfilling/interesting job outside of EA to become MIRI's janitor (if they'd be good).
I think your example would hold for someone donating a few hundred dollars to MIRI (which buys roughly 10^-4 additional researchers), without triggering the same ideas. Same goes for "contributing three useful LessWrong comments on posts about AI", "giving Superintelligence to one friend", etc. These examples are nice in that they also work for people who don't want to live in the Bay, are happy in their current jobs, etc.
Anyway, that's just a guess, which doubles as a critique of the shortform post. But I did upvote the post, because I liked this bit:
I wasn't sure where to add this comment, or idea rather. But I have something to propose hair that I think would make a gigantic impact on the well-being of millions. I don't have a fancy research or such to back it up yet but I'm sure there's money out there to support if needed. So here goes ... I remember learning about ancient times and Greek mythology and such and was very fascinated with the time period and a lot of it's heroes. I do believe that's back then four wars or sometimes fought by having two of the best warriors battled and place of whole a... (read more)
Here are some things I've learned from spending the better part of the last 6 months either forecasting or thinking about forecasting, with an eye towards beliefs that I expect to be fairly generalizable to other endeavors.
Note that I assume that anybody reading this already has familiarity with Phillip Tetlock's work on (super)forecasting, particularly Tetlock's 10 commandments for aspiring superforecasters.
1. Forming (good) outside views is often hard but not impossible. I think there is a common belief/framing in EA and rationalist circles that coming up with outside views is easy, and the real difficulty is a) originality in inside views, and also b) a debate of how much to trust outside views vs inside views.
I think this is directionally true (original thought is harder than synthesizing existing views) but it hides a lot of the details. It's often quite difficult to come up with and balance good outside views that are applicable to a situation. See Manheim and Muelhauser for some discussions of this.
2. For novel out-of-distribution situations, "normal" people often trust centralized data/ontologies more than is warranted. See here for a discu... (read more)
Consider making this a top-level post! That way, I can give it the "Forecasting" tag so that people will find it more often later, which would make me happy, because I like this post.
Recently I was asked for tips on how to be less captured by motivated reasoning and related biases, a goal/quest I've slowly made progress on for the last 6+ years. I don't think I'm very good at this, but I do think I'm likely above average, and it's also something I aspire to be better at. So here is a non-exhaustive and somewhat overlapping list of things that I think are helpful:
- Treat this as a serious problem that needs active effort.
- Personally, I repeat the mantra "To see the world as it is, not as I wish it to be" on a near-daily basis. You may have other mantras or methods to invoke this mindset that works better for you.
- In general, try to be more of a "scout" and less of a soldier/zealot.
- As much as possible, try to treat ideas/words/stats as tools to aid your perception, not weapons to advance your cause or "win" arguments.
- I've heard good things about this book by Julia Galef, but have not read it
- Relatedly, have a mental check and try to notice when you are being emotional and in danger of succumbing to motivated reasoning. Always try to think as coherently as you can if possible, and acknowledge it (preferably publicly) when you notice you're not.
- sometimes, you don't noti
... (read more)Is anybody trying to model/think about what actions we can do that are differentially leveraged during/in case of nuclear war, or the threat of nuclear war?
In the early days of covid, most of us were worried early on, many of us had reasonable forecasts, many of us did stuff like buy hand sanitizers and warn our friends, very few of us shorted airline stocks or lobbied for border closures or did other things that could've gotten us differential influence or impact from covid.
I hope we don't repeat this mistake.
A while ago, Spencer Greenberg asked:
I had a response that a few people I respect thought was interesting, so I'm reposting it here:
... (read more)A skill/attitude I feel like I improved a lot on in the last year, and especially in the last 3 months, is continuously asking myself whether any work-related activity or research direction/project I'm considering has a clear and genuine/unmovitated story for having a notable positive impact on the long-term future (especially via reducing x-risks), and why.
Despite its simplicity, this appears to be a surprisingly useful and rare question. I think I recommend more people, especially longtermism researchers and adjacent folks, to consider this explicitly, on a regular/almost instinctual basis.
Are there any EAA researchers carefully tracking the potential of huge cost-effectiveness gains in the ag industry from genetic engineering advances of factory farmed animals? Or (less plausibly) advances from better knowledge/practice/lore from classical artificial selection? As someone pretty far away from the field, a priori the massive gains made in biology/genetics in the last few decades seems like something that we plausibly have not priced in in. So it'd be sad if EAAs get blindsided by animal meat becoming a lot cheaper in the next few decades (if this is indeed viable, which it may not be).
Do people have advice on how to be more emotionally resilient in the face of disaster?
I spent some time this year thinking about things that are likely to be personally bad in the near-future (most salient to me right now is the possibility of a contested election + riots, but this is also applicable to the ongoing Bay Area fires/smoke and to a lesser extent the ongoing pandemic right now, as well as future events like climate disasters and wars). My guess is that, after a modicum of precaution, the direct objective risk isn't very high, but it'll *feel* like a really big deal all the time.
In other words, being perfectly honest about my own personality/emotional capacity, there's a high chance that if the street outside my house is rioting, I just won't be productive at all (even if I did the calculations and the objective risk is relatively low).
So I'm interested in anticipating this phenomenon and building emotional resilience ahead of time so such issues won't affect me as much.
I'm most interested in advice for building emotional resilience for disaster/macro-level setbacks. I think it'd also be useful to build resilience for more personal setbacks (eg career/relationship/impact), but I naively suspect that this is less tractable.
Thoughts?
What are the best arguments for/against the hypothesis that (with ML) slightly superhuman unaligned systems can't recursively self-improve without solving large chunks of the alignment problem?
Like naively, the primary way that we make stronger ML agents is via training a new agent, and I expect this to be true up to the weakly superhuman regime (conditional upon us still doing ML).
Here's the toy example I'm thinking of, at the risk of anthromorphizing too much:Suppose I'm Clippy von Neumann, an ML-trained agent marginally smarter than all humans, but nowhere near stratospheric. I want to turn the universe into paperclips, and I'm worried that those pesky humans will get in my way (eg by creating a stronger AGI, which will probably have different goals because of the orthogonality thesis). I have several tools at my disposal:
- Try to invent ingenious mad science stuff to directly kill humans/take over the world
- But this is too slow, another AGI might be trained before I can do this
- Copy myself a bunch, as much as I can, try to take over the world with many copies.
- Maybe too slow? Also might be hard to get enough resources to make more copies
- Try to persuade my human handlers to give me e
... (read more)No, you make a copy of yourself, do brain surgery on the copy, and copy the changes to yourself only if you are happy with the results. Yes, I think recursive improvement in humans would accelerate a ton if we had similar abilities (see also Holden on the impacts of digital people on social science).
I've been asked to share the following. I have not loved the EA communications from the campaign. However, I do think this is plausibly the most cost-effective use of time this year for a large fraction of American EAs and many people should seriously consider it (or just act on it and reflect later), but I have not vetted these considerations in detail.
[[URGENT]] Seeking people to lead a phone-banking coworking event for Carrick Flynn's campaign today, tomorrow, or Tuesday in gather.town ! There is an EA coworking room in gathertown already. This is a strong counterfactual opportunity! This event can be promoted on a lot of EA fb pages as a casual and fun event (after all, it won't even be affiliated with "EA", but just some people who are into this getting together to do it), hopefully leading to many more phone banker hours in the next couple days.
Would you or anyone else be willing to lead this? You (and other hosts) will be trained in phonebanking and how to train your participants in phonebanking.
Please share with people you think would like to help, and DM Ivy and/or CarolineJ (likely both as Ivy is traveling).
You can read more about Carrick's campaign from an EA... (read more)
While talking to my manager (Peter Hurford), I made a realization that by default when "life" gets in the way (concretely, last week a fair amount of hours were taken up by management training seminars I wanted to attend before I get my first interns, this week I'm losing ~3 hours the covid vaccination appointment and in expectation will lose ~5 more from side effects), research (ie the most important thing on my agenda that I'm explicitly being paid to do) is the first to go. This seems like a bad state of affairs.
I suspect that this is more prominent in me than most people, but also suspect this is normal for others as well. More explicitly, I don't have much "normal" busywork like paperwork or writing grants and I try to limit my life maintenance tasks (of course I don't commute and there's other stuff in that general direction). So all the things I do are either at least moderately useful or entertaining. Eg, EA/work stuff like reviewing/commenting on other's papers, meetings, mentorship stuff, slack messages, reading research and doing research, as well as personal entertainment stuff like social media, memes, videogames etc (which I do much more than I'm willing to admi... (read more)
I liked this, thanks.
I hear that this similar to a common problem for many entrepreneurs; they spend much of their time on the urgent/small tasks, and not the really important ones.
One solution recommended by Matt Mochary is to dedicate 2 hours per day of the most productive time to work on the the most important problems.
https://www.amazon.com/Great-CEO-Within-Tactical-Building-ebook/dp/B07ZLGQZYC
I've occasionally followed this, and mean to more.
So framing this in the inverse way – if you have a windfall of time from "life" getting in the way less, you spend that time mostly on the most important work, instead of things like extra meetings. This seems good. Perhaps it would be good to spend less of your time on things like meetings and more on things like research, but (I'd guess) this is true whether or not "life" is getting in the way more.
Thanks salius! I agree with what you said. In addition,
I think high probability of existential risk this century mostly dampens the force of fanaticism*-related worries/arguments. I think this is close to self-evident, though can expand on it if it's interesting.
*fanaticism in the sense of very small probabilities of astronomically high payoffs, not in the everyday sense of people might do extreme actions because they're ideological. The latter is still a worry.
I find the unilateralist’s curse a particularly valuable concept to think about. However, I now worry that “unilateralist” is an easy label to tack on, and whether a particular action is unilateralist or not is susceptible to small changes in framing.
Consider the following hypothetical situations:
- Company policy vs. team discretion
- Alice is a researcher in a team of scientists at a large biomedical company. While working on the development of an HIV vaccine, the team accidentally created an air-transmissible variant of HIV. The scientists must decide whether to publish their discovery with the rest of the company, knowing that leaks may exist, and the knowledge may be used to create a devastating biological weapon, but also that it could help those who hope to develop defenses against such weapons, including other teams within the same company. Most of the team thinks they should keep it quiet, but company policy is strict that such information must be shared with the rest of the company to maintain the culture of open collaboration.
- Alice thinks the rest of the team should either share this information or quit. Eventually, she tells her vice president her concer
... (read more)I'm worried about a potential future dynamic where an emphasis on forecasting/quantification in EA (especially if it has significant social or career implications) will have adverse effects on making people bias towards silence/vagueness in areas where they don't feel ready to commit to a probability forecast.
I think it's good that we appear to be moving in the direction of greater quantification and being accountable for probability estimates, but I think there's the very real risk that people see this and then become scared of committing their loose thoughts/intuitive probability estimates on record. This may result in us getting overall worse group epistemics because people hedge too much and are unwilling to commit to public probabilities.
See analogy to Jeff Kaufman's arguments on responsible transparency consumption:
https://www.jefftk.com/p/responsible-transparency-consumption
I think many individual EAs should spend some time brainstorming and considering ways they can be really ambitious, eg come up with concrete plans to generate >100M in moral value, reduce existential risk by more than a basis point, etc.
Likewise, I think we as a community should figure out better ways to help people ideate and incubate such projects and ambitious career directions, as well as aim to become a community that can really help people both celebrate successes and to mitigate the individual costs/risks of having very ambitious plans fail.
I'm now pretty confused about whether normative claims can be used as evidence in empirical disputes. I generally believed no, with the caveat that for humans, moral beliefs are built on a scaffolding of facts, and sometimes it's easier to respond to an absurd empirical claim with the moral claim that has the gestalt sense of empirical beliefs if there isn't an immediately accessible empirical claim.
I talked to a philosopher who disagreed, and roughly believed that strong normative claims can be used as evidence against more confused/less c... (read more)
I actually found this article surprisingly useful/uplifting, and I suspect reorienting my feelings towards this approach is both more impactful and more emotionally healthy for me. I think recently (especially in the last month), I was getting pretty whiny/upset about the ways in which the world feels unfair to me, in mostly not-helpful ways.
I know that a lot of the apparent unfairness is due to personal choices that I on reflection endorse (though am not necessarily in-the-moment happy about). However, I suspect there's something true and important about ... (read more)
Re the recent discussions on whether EAs are overspending on luxuries for ourselves, one thing that strikes me is that EA is an unusually international and online movement. This means that many of us will come from very different starting contexts in terms of national wealth, and/or differing incomes or social class. So a lot of the clashes in conceptions of what types of spending seems "normal" vs "excessive" will come from pretty different priors/intuitions of what seems "normal." For example, whether it's "normal" to have >100k salaries, whether it's "normal" to have conferences in fancy hotels, whether it's normal to have catered food, etc.
There are various different framings here. One framing that I like is that (for some large subset of funded projects) the EA elites often think that your work is more than worth the money. So, it's often worth recalibrating your expectations of what types of spending is "normal," and make adequate time-money tradeoffs accordingly. This is especially the case if you are seen as unusually competent, but come from a country or cultural background that is substantially poorer or more frugal than the elite US average.
Another framing that's worthwhile is to make explicit time-money tradeoffs and calculations more often.
Very instructive anecdote on motivated reasoning in research (in cost-effectiveness analyses, even!):
... (read more)In the Precipice, Toby Ord very roughly estimates that the risk of extinction from supervolcanoes this century is 1/10,000 (as opposed to 1/10,000 from natural pandemics, 1/1,000 from nuclear war, 1/30 from engineered pandemics and 1/10 from AGI). Should more longtermist resources be put into measuring and averting the worst consequences of supervolcanic eruption?
More concretely, I know a PhD geologist who's interested in doing an EA/longtermist career and is currently thinking of re-skilling for AI policy. Given that (AFAICT) literally zero people in our community currently works on supervolcanoes, should I instead convince him to investigate supervolcanoes at least for a few weeks/months?
If he hasn't seriously considered working on supervolcanoes before, then it definitely seems worth raising the idea with him.
I know almost nothing about supervolcanoes, but, assuming Toby's estimate is reasonable, I wouldn't be too surprised if going from zero to one longtermist researcher in this area is more valuable than adding an additional AI policy researcher.
crossposted from LessWrong
There should maybe be an introductory guide for new LessWrong users coming in from the EA Forum, and vice versa.
I feel like my writing style (designed for EAF) is almost the same as that of LW-style rationalists, but not quite identical, and this is enough to be substantially less useful for the average audience member there.
For example, this identical question is a lot less popular on LessWrong than on the EA Forum, despite naively appearing to appeal to both audiences (and indeed if I were to guess at the purview of LW, to be cl... (read more)
One thing I'd be excited to see/fund is a comprehensive survey/review of what being an independent researcher in EA is like, and what are the key downsides, both:
a. From a personal perspective. What's unpleasant, etc about the work.
b. From a productivity perspective. What are people missing in independent research that's a large hit to their productivity and/or expected impact in the world.
This is helpful for two reasons:
- More information/communication is generally helpful. I think having a clearer-eyed understanding of the downsides of independent research
... (read more)Something that came up with a discussion with a coworker recently is that often internet writers want some (thoughtful) comments, but not too many, since too many comments can be overwhelming. Or at the very least, the marginal value of additional comments is usually lower for authors when there are more comments.
However, the incentives for commentators is very different: by default people want to comment on the most exciting/cool/wrong thing, so internet posts can easily by default either attract many comments or none. (I think) very little self-policing is done, if anything a post with many comments make it more attractive to generate secondary or tertiary comments, rather than less.
Meanwhile, internet writers who do great work often do not get the desired feedback. As evidence: For ~ a month, I was the only person who commented on What Helped the Voiceless? Historical Case Studies (which later won the EA Forum Prize).
This will be less of a problem if internet communication is primarily about idle speculations and cat pictures. But of course this is not the primary way I and many others on the Forum engage with the internet. Frequently, the primary publication v... (read more)
I think it might be interesting/valuable for someone to create "list of numbers every EA should know", in a similar vein to Latency Numbers Every Programmer Should Know and Key Numbers for Cell Biologists.
One obvious reason against this is that maybe EA is too broad and the numbers we actually care about are too domain specific to specific queries/interests, but nonetheless I still think it's worth investigating.
I think I have a preference for typing "xrisk" over "x-risk," as it is easier to type out, communicates the same information, and like other transitions (e-mail to email, long-termism to longtermism), the time has come for the unhyphenated version.
Curious to see if people disagree.
I think it's really easy to get into heated philosophical discussions about whether EAs overall use too much or too little jargon. Rather than try to answer this broadly for EA as a whole, it might be helpful for individuals to conduct a few quick polls to decide for themselves whether they ought to change their lexicon.
Here's my Twitter poll as one example.
I'm curious if any of the researchers/research managers/academics/etc here a) read High Output Management b) basically believe in High Output Management's core ideas and c) have thoughts on how should it be adapted to research.
The core idea there is the production process:
"Deliver an acceptable product at the scheduled time, at the lowest possible cost."
The arguments there seem plausible and Andy Grover showed ingenuity in adapting the general idea into detailed stories for very different processes (eg running a restaraunt, training sales... (read more)
I've finally read the Huw Hughes review of the CE Delft Techno-Economic Analyses (our summary here) of cultured meat and thought it was interesting commentary on the CE Delft analysis, though less informative on the overall question of cultured meat scaleups than I hoped.
Overall their position on CE Delft's analysis was similar to ours, except maybe more bluntly worded. They were more critical in some parts and less critical in others.
Things I liked about the Hughes review:
- the monoclonal antibodies reference class point was interesting and not something I've considered before
- I liked how it spelled out the differences in stability between conventional meat and cell slurry; I've thought about it before but didn't seriously consider it, having this presented so clearly was useful
- I liked the author diagramming an entire concept area to see what a real factory might look like, instead of only looking at the myocyte cell process
- I fully agree with the author about the CE Delft TEA being way too underspecified, as well as the aseptic conditions stuff making food bioreactors a very unlikely reference class (though of course I already knew these points)
- I liked the points about reg
... (read more)On the forum, it appears to have gotten harder for me to do multiple quote blocks in the same comment. I now often have to edit a post multiple times so quoted sentences are correctly in quote blocks, and unquoted sections are not. Whereas in the past I do not recall having this problem?
cross-posted from Facebook.
Sometimes I hear people who caution humility say something like "this question has stumped the best philosophers for centuries/millennia. How could you possibly hope to make any progress on it?". While I concur that humility is frequently warranted and that in many specific cases that injunction is reasonable [1], I think the framing is broadly wrong.
In particular, using geologic time rather than anthropological time hides the fact that there probably weren't that many people actively thinking about these issues, especially carefully, in a sustained way, and making sure to build on the work of the past. For background, 7% of all humans who have ever lived are alive today, and living people compose 15% of total human experience [2] so far!!!
It will not surprise me if there are about as many living philosophers today as there were dead philosophers in all of written history.
For some specific questions that particularly interest me (eg. population ethics, moral uncertainty), the t