I have previously encountered EAs who have beliefs about EA communication that seem jaded to me. These are either, “Trying to make EA seem less weird is an unimportant distraction, and we shouldn’t concern ourselves with it” or “Sounding weird is an inherent property of EA/EA cause areas, and making it seem less weird is not tractable, or at least not without compromising important aspects of the movement.” I would like to challenge both of these views.
“Trying to make EA seem less weird is unimportant”
As Peter Wildeford explains in this LessWrong post:
People take weird opinions less seriously. The absurdity heuristic is a real bias that people -- even you -- have. If an idea sounds weird to you, you're less likely to try and believe it, even if there's overwhelming evidence.
Being taken seriously is key to many EA objectives, such as growing the movement, getting mainstream researchers to care about the EA risks of their work, and having policymakers give weight to EA considerations. Sci-fi-sounding ideas make it harder to achieve these, while making it easier for critics to mischaracterize the movement and (probably) contributing to the perception that EA is cult-like.
On a more personal note, and perhaps more relevant to some of the examples I am going to mention, it is also nice for friends and family to be able to understand why we do what we do, which I don’t think is a trivial desire to have.
All things considered, I think it would be better if EA ideas did not sound weird to outsiders, and instead sounded intuitive and immediately persuasive.
“EA is inherently weird, and making our ideas seem less weird is not tractable”
I don’t think this is true, and I think this view generally comes from people who haven’t spent much time trying to think creatively about this. Some ways of framing things are more compelling than others, and this is an area where we can iterate, innovate and improve. Here are a few examples of possible ways we talk about weird EA ideas:
AI risk
Talking about other bad but less severe outcomes of AI misalignment besides paperclip maximizers and then saying “and it could even get as bad as paperclip maximizers,” requires less of a leap in imagination than opening with paperclip maximizers. It may be the case that we don’t even need to make general audiences consider paperclip maximizers at all, since the mechanisms needed to prevent them are the same as those needed to prevent the less severe and more plausible-sounding scenarios of the form "you ask an AI to do X, and the AI accomplishes X by doing Y, but Y is bad and not what you intended".
Longtermism
Due to scope insensitivity, referencing visuals to show just how much larger the future can be than the present is particularly emotionally powerful here, and makes the whole idea of working to improve the far future feel far less abstract. My favorite longtermist visualization is this one by Our World in Data, which I have saved on my phone to be able to reference it in conversations. (I think visualizations also work well to combat scope insensitivity for wild animal welfare and farmed animal welfare).
Non-human Welfare
If it is the first time someone is contemplating the idea that insects or wild animals deserve moral consideration, it makes sense to want to give them the spiel with the least probability of being mocked and dismissed. If you start to explain it by saying we should spend money to enhance the lives of insects in the wild, the idea will probably get laughed out of the room.
I think for insect welfare, the most palatable approach would be talking about the role of inhumane pesticides and other ways humans harm insects actively, making insect welfare more comparable to farmed animal welfare than wild animal welfare. Similarly, talking about helping wild animals during pandemics, famines, wildfires, etc. (problems humans also have) probably incites more compassion than talking about helping them from being chased by lions. How sensible something sounds to a layperson seems correlated with how tractable it is, so tractability can be used as a proxy for how likely an idea is to be dismissed by the person you are talking to.
The point is not to commit the motte-and-bailey fallacy (and one must be careful not to do this), but that people will be more open to contemplating your idea if you go in motte first instead of bailey first.
Other existential risks
I think the point about paperclip maximizers generalizes—it is sometimes not necessary to frame existential risks as existential risks. Most still have unusually high expected value even if they fall short of extinction, and this can be a preferable framing in some cases. Extinction-level events can be difficult to imagine and emotionally process, leading to overwhelm and inaction (see climate paralysis). We can say that serious pandemics are one of the “highest priority risks” for the international community due to their potential to kill hundreds of millions of people, and in many cases this would resonate more than the harder-to-conceive “existential risk” that could “lead to the extinction of humanity”. (Whether a problem poses extinction risk is, of course, still a relevant factor for cause prioritization.)
Also, as has been discussed many times already, longtermism is not a necessary prerequisite to care about existential risk. The expected value in the short term is enough to make people care about it, so trying to pitch existential risk through longtermism requires convincing people of an extra weird step and unnecessarily makes it less compelling to most.
Conclusion
My point is not necessarily that we should implement these specific examples, but that there are ways we can make our ideas more palatable to people. Also, there is the obvious caveat that the best way to talk about a topic will depend on your audience, but that doesn’t mean there aren’t some ways of communicating that work better most of the time, or work better for the general public.
Edit: the "AI accomplishes X by doing Y" thing I was talking about is called specification gaming, and here are some examples of it
I agree with the main point that we could sound less weird if we wanted to, but it seems unlikely to me that we want that.
This is just not true.
If you convince someone of a different, non-weird version of AI risk, that does not then mean that they should take the actions that we take. There are lots of other things you can do to mitigate the less severe versions of AI risk:
Most of these things are slightly helpful but overall don't have much effect on the versions of AI risk that lead to extinction.
(I expect this to generalize beyond AI risk as well and this dynamic is my main reason for continuing to give the weird version of EA ideas.)
This does seem to be an important dynamic.
Here are a few reasons this might be wrong (both sound vaguely plausible to me):
(1) is particularly important if you think this "non-weird to weird" approach will appeal to a set of people who wouldn't otherwise end up agreeing with your arguments. That would mean it has a high counterfactual impact - even if some of the people do work that whilst still being good is ultimately far less relevant to x-risk reduction. This is even more true if you think there's a low rate of people who would have just listened to your weirder sounding arguments in the first place who will get "stuck" at the non-weird stuff and as a result never do useful things.
I agree with both of those reasons in the abstract, and I definitely do (2) myself. I'd guess there are around 50 people total in the world who could do (2) in a way where I'd look at it and say that they succeeded (for AI risk in particular), of which I could name maybe 20 in advance. I would certainly not be telling a random EA to make our arguments sound less weird.
(EDIT: My estimate is now more like 30; I actually asked some people to do (2) for AI alignment and they did worse than I expected.)
I'd be happy about the version of (1) where the non-weird version was just an argument that people talked about, without any particular connection to EA / AI x-risk. I would not say "make EA sound less weird", I'd say "one instrumental strategy for EA is to talk about this other related stuff".
I strongly agree that you want to avoid EA jargon when doing outreach. Ideally you would use the jargon of your audience, though if you're talking to a broad enough audience that just means "plain English".
I disagree that "sounding weird" is the same thing (or even all that correlated with) "using jargon". For example,
This has no jargon, but still sounds weird to a ton of people. Similarly I think with AI risk the major weird part is the thing where the AI kills all the humans, which doesn't seem to me to depend much on jargon.
(If anything, I've found that with more jargon the ideas actually sound less weird. I think this is probably because the jargon obscures the meaning and so people can replace it with some different less weird meaning and assume you meant that. If you say "a goal-directed AI system may pursue some goal that we don't want leading to a catastrophic outcome" they can interpret you as saying "I'm worried about AIs mimicking human biases"; that doesn't happen when you say "an AI system may deliberately kill all the humans".)
That was a really clarifying reply!
tl;dr
I intuitively see vocabulary (jargon v. plain English) and framing as pretty closely related (there seems to almost be a continuum between "an entirely foreign language" to "extremely relatable and familiar language and framing of an idea"), so I conflated them a lot in my previous comment. They aren't the same though and it's definitely less of a red flag to me if someone can't find a relatable framing than if they can't put their point into plain English.
I think there is a clear mechanism for less jargon to make ideas sound less weird. When you find a way to phrase things in a more familiar way (with more familiar language; i.e. more familiar vocabulary, phrasing, framings, metaphors etc), the idea should sound more familiar and, therefore, less weird (so finding a more normal, jargon-free way of saying something should make it sound less weird).
However, I can also see there is also a mechanism for weird ideas to sound more weird with less jargon. If the idea is weird enough, then being more clear by using more familiar language will often make the idea sound weirder.
If there is no perfect "translation" of an idea into the familiar, then I agree that it's better to actually communicate the idea and sound weird than to sound less weird due to creating a false impression.
"an AI system may deliberately kill all the humans" sounds really weird to most people partly because most people don't have the context to make sense of the phrase (likewise with your other plain English example). That missing context, that inferential gap, is a lot like a language barrier.
The heuristic "make yourself sound less weird" seems helpful here.
I think it sounds a lot less weird to say "an AI system might be hard to control and because of that, some experts think it could be really dangerous". This doesn't mean the same thing as "an AI might kill us all", but I think it lays the groundwork to build the idea in a way that most people can then better contextualize and understand more accurately from there.
Often, the easiest way to make EA ideas sound less weird is just to break up ideas into smaller pieces. Often EA ideas sound weird because we go way too fast and try to give way more information than a person can actually process at once. This is not actually that helpful for leaving accurate impressions. They might technically know the conclusion, but without enough steps to motivate it, an isolated, uncontextualized conclusion is pretty meaningless.
I think the advice "try to sound less weird" is often a helpful heuristic. You've convinced me that a caveat to the advice is maybe needed: I'd much rather we were weird than misleading!
What do you think of a caveated version of the advice like "make EA ideas sound less weird without being misleading"? Do you think this caveated version, or something like it, is better than no advice?
I mostly agree with this.
I think it sounds a lot less weird in large part because you aren't saying that the AI system might kill us all. "Really dangerous" could mean all sorts of things, including "the chess-playing robot mistakes a child's finger for a chess piece and accidentally breaks the child's finger". Once you pin it down to "kills all humans" it sounds a lot weirder.
I still do agree with the general point that as you explain more of your reasoning and cover more of the inferential gap, it sounds less weird.
I still worry that people will not realize the ways they're being misleading -- I think they'll end up saying true but vague statements that get misinterpreted. (And I worry enough that I feel like I'd still prefer "no advice".)
tl;dr:
Imagine a person at MIT comes to an EA society event and has some conversations about AI and then never comes back. Eventually, they end up working at Google and making decisions about Deepmind's strategy.
Which soundbite do I want to have given them? What is the best 1 dimensional (it's a quick conversation, we don't have time for the 2 dimensional explanation) message I could possibly leave this person with?
Option 1: "an AI might kill us all" (they think we think a Skynet scenario is really likely and we have poor reasoning skills because a war with walking robots is not that reasonable)
Option 2: "an AI system might be hard to control and because of that, some experts think it could be really dangerous" (this statement accurately applies to the "accidentally breaks the child's finger" case and also world-ending scenarios, in my mind at least, so they've fully understood my meaning even if I haven't yet managed to explain my personal bottom line)
I think they will be better primed to make good decisions about safe AI if I focus on trying to convey my reasoning before I try and communicate my conclusion. Why? My conclusion is actually not that helpful to a smart person who wants to think for themselves without all the context that makes that conclusion reasonable. If I start with my reasoning, even if I don't take this person to my bottom line, someone else down the road who believes the same thing as me can take them through the next layer up. Each layer of truth matters.
If it sounds weird, it's probably because I've not given enough context for them to understand the truth and therefore I haven't really done any good by sharing that with them (all I've done is made them think I'm unreasonable).
My guess is this person who came to one EA event and ended up being a key decisionmaker at Deepmind is going to be a lot less resistant when they hear about AI alignment in their job if they heard option 2 and not option 1. Partly because the groundwork to the ideas was better laid out. Partly because they trust the "effective altruism" brand more because they have the impression the effective altruism people, associated with "AI alignment" (an association that could stick if we keep going the way we've been going), to be full of reasonable people who think reasonable things.
What matters is whether we've conveyed useful truth, not just technically true statements. I don't want us to shy away from communicating about AI, but I do want us to shy away from communicating about AI in a confronting way that is counterproductive to giving someone a very nuanced understanding later on.
I think the advice "make AI sound less weird" is better than no advice because I think that communicating my reasoning well (which won't sound weird because I'll build it up layer by layer) is more important than communicating my current bottom line (leaving an impression of my bottom line that has none of the context attached to make it meaningful, let alone nuanced and high-fidelity) as quickly as possible.
PS: I still don't think I've actually done a good job of laying the reasoning for my views clearly here so I'm going to write a post at some point (I don't have time to fix the gaps I see now). It is helpful for you to point out gaps you see out explicitly so I can fill them in future writing if they actually can be filled (or change my mind if not).
In the meantime, I wanted to say that I've really valued this exchange. It has been very helpful for forcing me to see if I can make my intuitions/gut feeling more explicit and legible.
I agree that if the listener interprets "make EA sound less weird" as "communicate all of your reasoning accurately such that it leads the listener to have correct beliefs, which will also sound less weird", then that's better than no advice.
I don't think that's how the typical listener will interpret "make EA sound less weird"; I think they would instead come up with surface analogies that sound less weird but don't reflect the underlying mechanisms, which listeners might notice then leading to all the problems you describe.
I definitely don't think we should just say all of our conclusions without giving our reasoning.
(I think we mostly agree on what things are good to do and we're now hung up on this not-that-relevant question of "should we say 'make EA sound less weird'" and we probably should just drop it. I think both of us would be happier with the advice "communicate a nuanced, accurate view of EA beliefs" and that's what we should go with.)
Note: edited significantly for clarity the next day
Tl;dr: Weirdness is still a useful sign of sub-optimal community building. Legibility is the appropriate fix to weirdness.
I know I used the terms "nuanced" and "high-fidelity" first but after thinking about it a few more days, maybe "legibility" more precisely captures what we're pointing to here?
Me having the hunch that the advice "don't be weird" would lead community builders to be more legible now seems like the underlying reason I liked the advice in the first place. However, you've very much convinced me you can avoid sounding weird by just not communicating any substance. Legibility seems to capture what community builders should do when they sense they are being weird and alienating.
EA community builders probably should stop and reassess when they notice they are being weird, "weirdness" is a useful smoke alarm for a lack of legibility. They should then aim to be more legible. To be legible, they're probably strategically picking their battles on what claims they prioritize justifying to newcomers. They are legibly communicating something, but they're probably not making alienating uncontextualized claims they can't back-up in a single conversation.
They are also probably using clear language the people they're talking to can understand.
I now think the advice "make EA more legible" captures the upside without the downsides of the advice "make EA sound less weird". Does that seem right to you?
I still agree with the title of the post. I think EA could and should sound less weird by prioritizing legibility at events where newcomers are encouraged to attend.
Noticing and preventing weirdness by being more legible seems important as we get more media attention and brand lock-in over the coming years.
Yeah I'm generally pretty happy with "make EA more legible".
Cool . I'm curious, how does this feeling change for you if you found out today that AI timelines are almost certainly less than a decade?
I'm curious because my intuitions change momentarily whenever a consideration pops into my head that makes me update towards AI timelines being shorter.
I think my intuitions change when I update towards shorter AI timelines because legibility/the above outlined community building strategy has a longer timeline before the payoffs. Managing reputation and goodwill seem like good strategies if we have a couple of decades or more before AGI.
If we have time, investing in goodwill and legibility to a broader range of people than the ones who end up becoming immediately highly dedicated seems way better to me.
Legible high-fidelity messages are much more spreadable than less legible messages but they still some take more time to disseminate. Why? The simple bits of it sound like platitudes. And the interesting takeaways require too many steps in logic from the platitudes to go viral.
However, word of mouth spread of legible messages that require multiple steps in logic still seem like they might spread exponentially (just with a lower growth rate than simpler viral messages).
If AI timelines are short enough, legibility wouldn't matter in those possible worlds. Therefore, if you believe timelines are extremely short then you probably don't care about legibility or reputation (and you also don't advise people to do ML PhDs because by the time they are done, it's too late).
Does that seem right to you?
Idk, what are you trying to do with your illegible message?
If you're trying to get people to do technical research, then you probably just got them to work on a different version of the problem that isn't the one that actually mattered. You'd probably be better off targeting a smaller number of people with a legible message.
If you're trying to get public support for some specific regulation, then yes by all means go ahead with the illegible message (though I'd probably say the same thing even given longer timelines; you just don't get enough attention to convey the legible message).
TL;DR: Seems to depend on the action / theory of change more than timelines.
Goal of this comment:
This comment fills in more of the gaps I see that I didn't get time to fill out above. It fleshes out more of the connection between the advice "be less weird" and "communicate reasoning over conclusions".
I suspect that this is what most community builders will lay the groundwork to more legibly support conclusions when given advice like "get to the point if you can, don't beat around the bush, but don't be weird and jump the gun and say something without the needed context for the person you are talking to to make sense of what you are saying"
I feel like making arguments about stuff that is true is a bit like sketching out a maths proof for a maths student. Each link in the chain is obvious if you do it well, at the level of the person with whom you are taking through the proof, but if you start with the final conclusion, they are completely lost.
You have to make sure they're with you every step of the way because everyone gets stuck at a different step.
You get away with stating your conclusion without the proof in maths because there is a lot of trust that you can back up your claim (the worst thing that happens is the person you are talking to loses confidence in their ability to understand maths if you start with the conclusion before walking them through it at a pace they can follow).
We don't have that trust with newcomers until we build it. They won't suspect we're right unless we can show we're right in the conversation we made the claim in.
They'll lose trust and therefore interest very fast if we make a claim that requires at least 3 months of careful thought to come to a nuanced view on it. AI risk takes a tonne of time to develop inside views on. There is a lot of deference because it's hard to think the whole sequence through yourself for yourself and explore various objections until you feel like it's your view and not just something dictated to you. Deference is weird too (and gets a whole lot less weird when you just admit that you're deferring a bit and what exactly made you trust the person you are deferring to to come to reasonable views in the first place).
I feel like "don't sound weird" ends up translating to "don't say things you can't backup to the person you are talking to". In my mind, "don't sound weird" sounds a lot like "don't make the person you are talking to feel alienated", which in practice means "be legible to the person you are talking to".
People might say much less when they have to make the person they are talking to understand all the steps along the way, but I think that's fine. We don't need everyone to get to the bottom line. It's also often worse than neutral to communicate the bottom line without everything above it that makes it reasonable.
Ideally, community builders don't go so glacially slowly that they are at a standstill, never getting to any bottom lines that sound vaguely controversial, but while we've still got a decent mass of people who know the bottom line and enough of the reasoning paths that can take people there, it seems fine to increase the number of vague messages in order to decrease the number of negative impressions.
I still want lots of people who understand the reasoning and the current conclusions, I just don't think starting with an unmotivated conclusion is the best strategy for achieving this and I think "don't be weird" plus some other advice to stop community builders from literally stagnating and never getting to the point seems much better than the current status quo.
Fair enough.
tl;dr: I now think that EA community-builders should present ideas in a less weird way when it doesn't come at the expense of clarity, but maybe the advice "be less weird" is not good advice because it might make community-builders avoid communicating weird ideas that are worth communicating.
You probably leave some false impressions either way
In some sense (in the sense I actually care about), both statements are misleading.
I think that community builders are going to convey more information, on average, if they start with the less weird statement.
Often inferential gaps can't be crossed in a conversation.
That missing understanding will always get filled in with something inaccurate (if they had an accurate impression, then there is no inferential gap here). The question is, which misconceptions are better to leave someone with?
You've outlined how "an AI system might be hard to control and because of that, some experts think it could be really dangerous" could be misunderstood. I agree that people are unlikely to think you mean "the AI system will kill us all" without further elaboration. They will attach more accessible examples to the vaguer statement in the meantime. It is unlikely they will attach one really specific wrong example though, there is ambiguity there and the uncertainty left from that ambiguity is much better than a strongly held false impression (if those are the two choices, ideally, the inferential gap gets closed and you get a strongly held true impression).
People who are hearing the statement "the AI system will kill us all" without further context will still try and attach the most accessible examples they have to make the phrase make as much sense as possible to them. This tends to mean Skynet style walking robots. They'll also probably hypothesize that you don't have very good epistemics (even if this is not the language they'd use to describe it). They won't trust you to have good reasons to believe what you do because you've made an extraordinary claim without having laid out the case for it yet. These are false impressions too. They are also likely to stick more because the extra weirdness makes these first impressions much more memorable.
Which impression do I prefer community builders leave newcomers with?
I value community builders conveying the reasoning processes much more than the bottom line. I want newcomers to have the tools to come to reasonable conclusions for themselves (and I think giving newcomers the reasons why the EA community has its current conclusions is a good start).
Giving newcomers a more accurate impression of a conclusion without giving them much context on that conclusion seems often worse than nothing. Especially since you often lose the trust of reasonable people when you make very surprising claims and can't back them up in the conversation (because that's too much of an inferential distance to cross in the time you have).
Giving them an accurate impression of one of the reasons for a conclusion seems neutral (unaligned AI seems like it could be an x-risk because AI systems are hard to control). That isolated reason without further elaboration doesn't actually say that much, but I think it does lay the groundwork for a deeper understanding of the final conclusion "AI might kill us all" if future conversations happen down the road.
My takeaways
After this discussion, I've changed my mind on "be less weird" being the right advice to get what I want. I can see how trying to avoid being weird might make community builders avoid getting the point across.
Something like "aim to be as accurate as possible using language and examples the person you are talking to can understand" still probably will result in less weird. I'd be surprised if it resulted in community builders obscuring their point.
I liked this comment!
In particular, I think the people who are good at "not making EA seem weird" (while still communicating all the things that matter – I agree with the points Rohin is making in the thread) are also (often) the ones who have a deeper (or more "authentic") understanding of the content.
There are counterexamples, but consider, for illustration, that Yudkowsky's argument style and the topics he focuses on would seem a whole lot weirder if he wasn't skilled at explaining complex issues. So, understanding what you talk about doesn't always make your points "not weird," but it (at least) reduces weirdness significantly.
I think that's mostly beneficial and I think fewer people coming into contact with EA ideas but where they do come into contact with them, they hear them from exponents with a particularly deep, "authentic" understanding, seems like a good thing!
Instead of (just) "jargon" you could also say "talking points."
tl;dr:
I completely agree that you could replace "jargon" with "talking points".
I also agree with Rohan that it's important to not shy away from getting to the point if it is possible you can make the point in a well-reasoned way.
However, I actually think it's possibly quite important for improving the epistemics of people new to the community for there to be less pressure to communicate "all the things that matter". At least, I think there needs to be less pressure to communicate all the things that matter all at once.
The sequences are long for a reason. Legible, clear reasoning is slow. I think too much pressure to get to every bottom line in a very short time makes people skip steps. This means that not only are we not showing newcomers what good reasoning processes look like, we are going to be off-putting to people who want to think for themselves and aren't willing to make huge jumps that are missing important parts of the logic.
Pushing community builders to get to all the important key points, many bottom lines, will maybe make it hard for newcomers to feel like they have permission to think for themselves and make their own minds up. To feel rushed to a conclusion, to feel like you must come to the same conclusion as everyone else, no matter how important it is, will always make clear thinking harder.
If we want a community full of people who have good reasoning processes, we need to create environments where good reasoning processes can thrive. I think this, like most things, is a hard trade-off and requires community builders to be pretty skilled or to have much less asked of them.
If it's a choice between effective altruism societies creating environments where good reasoning processes can occur or communicating all the bottom lines that matter, I think it might be better to focus on the former. I think it makes a lot of sense to have effective altruism societies to be about exploration.
We still need people to execute. I think having AI risk specific societies, bio-risk societies, broad longtermism societies, poverty societies (and many other more conclusion focused mini-communities) might help make this less of a hard trade-off (especially as the community grows and there becomes more room for more than one effective altruism related society on any given campus). It is much less confusing to be rushed to a conclusion when that conclusion is well-labelled from the get-go (and effective altruism societies then can point interested people in the right direction to find out why certain people think certain bottom lines are sound).
Whatever the solution, I do worry rushing people to too many bottom lines too quickly does not create the community we want. I suspect we need to ask community builders to communicate less (we maybe need to triage our key points more), in order for them to communicate those key points in well-reasoned way.
Does that make sense?
Also, I'm glad you liked my comment (sorry for writing an essay objecting to a point made in passing, especially since your reply was so complementary; clearly succinctness is not my strength so perhaps other people face this trade-off much less than me :p).
I think there are trade-offs here. Ideally, I think we want community builders to prioritise high-fidelity, nuanced and precise communication over appealing to more people, but we also want community builders to prioritise broader appeal over signalling they are in the in-group or that they are intelligent (we’re all human and I think it takes conscious effort to push back against these instincts to 1) make it clear you belong in the tribe by signalling in-groupness and 2) that you possess traits that are valued by the tribe like intelligence)
Relevant example 😅🤣🤮😿