Can you clarify what your assumptions are about the criticized orgs? What would an org have to do before it became a bad idea to follow these steps?
It seems like these steps rest on an assumption of the org both wanting to be cooperative and having the skill to do it well. And not, e.g. explicitly promising confidentiality and then forwarding it to the exact person the author didn't want to see it. I believe that was an honest, human mistake, but that's exactly the problem: orgs are human- er, made up of humans- who have full plates and stuff gets missed. If organizations want the favor of previewing criticism, they need to take reasonable steps to make that easy and productive for critics.
One particular way to do this would be to talk with people earlier in the process. The discussion I've seen around previewing criticism implicitly assumes there's a full polished blog post, ready to go except for the criticized org's response. That's a very annoying time to find entirely new facts, and imposes a very high tax on criticism. It also makes it easy for a defensive or merely human org to create a bunch more work for the critic without adding value.
If orgs made themselves available earlier in the process, to discuss concerns before someone invested tens of hours doing their own research and carefully writing it up, I would feel much better about strong norms of checking in with them ahead of time. Alas this probably necessitates spending a bunch of time with potential critics whose criticisms are stupid or who were never going to write them, which is pretty costly. But so is spending 30 hours writing a delicate, polished piece of criticism only to have the org create a bunch of work for you.
In this post the criticizer gave the criticizee an opportunity to reply in-line in the published post—in effect, the criticizee was offered the last word. I thought that was super classy, and I’m proud to have stolen that idea on two occasions (1,2).
If anyone’s interested, the relevant part of my email was:
…
You can leave google docs margin comments if you want, and:
If I’m just straight-up wrong about something, or putting words in your mouth, then I’ll just correct the text before publication.
If you are leave a google docs comment that’s more like a counter-argument, and I’m not immediately convinced, I’d probably copy what you wrote into an in-text reply box—just like the gray boxes here: [link] So you get to have the last word if you want, although I might still re-reply in the comments.
You can also / alternatively obviously leave comments on the published lesswrong post like normal.
If you would like to leave pre-publication feedback, but don’t expect to get around to it “soon” (say, the next 3 weeks), let me know and I’ll hold off publication.
(In the LW/EAF post editor, the inline reply-boxes are secretly just 1×1 tables.)
Another super classy move was I wrote a criticism post once, and the person I criticized retweeted it. (Without even dunking on it!) (The classy person here was Robin Hanson.) I’m proud to say that I’ve stolen that one too, although I guess not every time.
Following up on this note: below is a non-exhaustive list of other potentially-relevant assumptions that I won’t bother discussing but that I might rely on (to different degrees) in this post:
Criticism of someone’s work is more likely than other kinds of critical writing (like disagreement with someone’s written arguments) to be wrong or misleading because of an information asymmetry. It’s pretty common for criticism to be at least somewhat misleading (even if it still has significant and useful points).
When you’re writing about work that isn’t an entirely public output (communication/research that everyone can access), you’re more likely to simply lack context or be wrong about what you’re writing about. You can’t just reference specific parts of the work; the person or people who’ve done the work know things that you don’t.
It’s useful to help people orient towards criticism of their work in healthy and positive ways, which can mean trying to make the process less stressful for them. This isn’t against the critical spirit or the like.
Wrong or somewhat misleading criticism of people’s work can be pretty harmful, especially if a response from those criticized doesn’t come right away.
Readers come away with incorrect (and extremely negative) opinions of the work being criticized.
It can be harder for readers to tell for themselves whose side or claims they should believe than if this were a disagreement about public content (similar to the dynamics outlined in Assumption 1); they, too, lack information and will be potentially more susceptible to believing one side or the other based on who sounds more convincing or based on biases they already have.
Readers could interpret the criticism as a broad and very negative judgement of all the work ever done by the relevant people, even if it’s not meant that way.
If a lot of readers start incorrectly believing that someone’s work is bad in certain ways, this can harm that person’s ability to do other work.
It’s really stressful when someone shares incorrect criticism of your work publicly, especially if you don’t feel like you have the chance to defend yourself in time for people to see your counterpoints.
Stress or negative experiences like this lead people to be overall more negative about criticism, even when it’s accurate, productive, kindly presented, etc.
Responses from the people criticized are worth showing alongside the criticism even if we think that the criticism is on point. In particular, it's useful to give them the chance to write them in time to post an early comment on your post.
Showing responses alongside criticism lets readers form more independent opinions.
Showing responses like this helps make the criticism feel more collaborative.
People writing criticism in EA are often in a collaborative relationship with the people whose work they’re criticizing; this isn’t a zero-sum relationship, they have common goals, they don’t want to cause unnecessary harm, etc..
People who have less context on a project or who are missing significant information can still have incredibly useful perspectives and suggestions to contribute.
Maybe the work could be improved in a way you have special experience with, or you’ve spotted mistakes others missed — the people working on a project could be biased, and not tracking relevant information, etc.
Criticism should not be suppressed.
Indefinite or extended delays (of criticism and of other stuff) are pretty dangerous and can result in ~silent vetoes. They should be avoided.
People doing work in EA are humans whose feelings matter and can be helped in productive ways.
Appendix 2. Where I personally think the norm should be on whether to run criticism past people
I think it’s generally good to give people a heads-up if you’re writing a post criticizing their work (and I tentatively think a week or two is a reasonable timeline).
I don’t think we need to push people to be willing to update drafts if they get a private response to their draft before the criticism is posted publicly. I know of multiple cases where the critic got pulled into an extended back-and-forth as a result of doing something like this, and I suspect that this kind of thing sometimes causes people to just not post anything in cases where something should have been posted. (There are still clear benefits to being willing to update drafts, but there are also real costs.)
I also think that running criticism past the people whose work is being criticized should be a soft norm that you can override if your criticism is time-sensitive (e.g. there’s something happening tomorrow that you want to inform the Forum audience about) or if you have good reason to believe that people don’t share your fundamental values, are not acting in good faith, might pressure you to stop you from publishing, or would react in a hostile way for one reason or another. (E.g. if someone were to write a criticism about how MacDonalds shouldn’t be serving meat, I don’t think they should feel obliged to first get in touch with MacDonalds. Jeff notes another example in his “EDIT” here.)
In particular, from a moderation perspective: we will not ban people from the Forum for posting criticism without running it past anyone.
Note: if you’re thinking of sharing very personal information or something closer to an unverified rumor, I’d more strongly push you to run it past people in advance. You can check with the moderation team if you’re not sure whether something is personal. See also the policies on revealing personal information on the Forum.
Appendix 1.[1] Reasons to run criticism past the people whose work you’re criticizing, and reasons to not do it
Please feel free to suggest more!
Reasons to run it past the people whose work you’re criticizing (the “doers”):
The “doers” get a heads-up and don’t have to scramble to pull together a response (explaining where they disagree or how they see things, or just thanking you, etc.), and readers might come away better informed as a result.
The timeliness of responses to public criticism sometimes matters; readers might not see a response posted later. So when criticism is posted without a heads-up, the people who were criticized often drop what they were doing to respond, which often has extra costs.
If the response is delayed and the criticism you've shared is even slightly off, readers who don’t see a response posted later might come away with incorrect impressions of the work being criticized.
Responses like this often take time, especially if multiple people need to discuss the response (e.g. because facts are spread out across different people, or if there’s a legal issue).
Scrambling can also be unnecessarily stressful.
You might improve your criticism before you publish it.
It’s pretty easy to accidentally write something incorrect or misleading when criticizing someone’s work.
If they respond, you might discover that:
Your criticism was premised on something incorrect, or missing information. I think it would be interesting to publish a null result in that situation: “I thought that [X bad thing was happening], but it turns out that [something else].”
There’s information you were missing (or other considerations), but you still disagree on something important. Your criticism might become a disagreement about tradeoffs or something else.
Running criticism past the people whose work is criticized supports a more cooperative environment between people writing on the Forum and people doing stuff that could be criticized in EA — and I think that this cooperative environment leads to many other good things.
When you give “doers” a heads up, they might feel more like you’re working with them. When this is the norm, “doers” in EA are less likely to worry that someone might criticize them at any moment based on a misconception or based on how something looks (which is stressful!).
They’re more likely to react positively to criticism in the future.
Readers see that you ran your criticism past the “doers” and see the doers’ response, which can lead to a sense that the criticism is collaborative, and a reframing of the criticism as help for the project that is being criticized, rather than a holistic indictment of the doers. (If that’s the case.)
Reasons to NOT run it past the people whose work you're criticizing:
If you expect them to try to pressure you into not posting your criticism, you might want to avoid sharing it with them in advance.
If this happens, you might also want to consider getting in touch with someone.
It might add too much work for you.
It will probably delay the publication of your criticism a bit, and that’s a real cost. If your criticism is time-sensitive, you might not want to wait.
You might worry that you’ll get dragged into an extended back-and-forth, and either have to cut it off abruptly in a way that might be stressful, or waste a lot of time and energy.
If you have absolutely no reason to believe that the people whose work you’re criticizing are acting in good faith, you might not bother reaching out to them first.
A strong norm of doing this can have negative effects on the network, and doing it can reinforce the norm.
One of the significant negative effects a nomr like this might have is that people might feel like they have to do this, and if they don’t want to, that might lead to some criticisms never getting published.
I think this is the strongest point against having this as a norm.
There are some other effects. For instance, the definition of “criticism” is pretty vague, and maybe we'd end up with some odd misrepresentations of content if we support this norm. (E.g. people asking insinuating questions instead of straightforwardly criticizing because they don’t want their content to be labeled as criticism.) Or maybe some criticisms get too softened, and readers won’t realize that the author is trying to say that a project is totally counterproductive.
These are pretty rough notes that I pulled out to make the main post easier to use as a guide. The comment format is somewhat awkward, but it might work — I'm curious to hear how people feel about it.
The email of the person can usually be inferred from the name, and domain of the website of the organisation. For example, to contact you (Lizka Vaintrob) who work at CEA (whose website domain is centreforeffectivealtruism.org), I would consider the following (in this order):
“ Are you up for modifying your draft based on private responses from the people whose work you're criticizing?”
If you’ve solicited feedback, I would suggest that your obligation to modify your draft depends on how strong their response is.
If it’s clear that your draft contains significant factual inaccuracies, then publishing reflects poorly on you (though you shouldn’t automatically feel a need to respond to attempts to dispute a fact, as often they may be pointing out something minor or you may have reason to doubt their account).
If the average reader after reading both your post and their response will come away with an impression that your critique was ideological or one-sided, then you should probably edit your draft, though there might be exceptions (say if someone is traumatised to the point where they lack the ability to be objective, so the alternative would be not publishing at all).
To be clear, I’m not saying that you need to edit just because they’ve made some good or reasonable points. I’m more suggesting that you shouldn’t ignore points if this would make a good faith reader feel like your article was pushing them away from the truth.
Can you clarify what your assumptions are about the criticized orgs? What would an org have to do before it became a bad idea to follow these steps?
It seems like these steps rest on an assumption of the org both wanting to be cooperative and having the skill to do it well. And not, e.g. explicitly promising confidentiality and then forwarding it to the exact person the author didn't want to see it. I believe that was an honest, human mistake, but that's exactly the problem: orgs are human- er, made up of humans- who have full plates and stuff gets missed. If organizations want the favor of previewing criticism, they need to take reasonable steps to make that easy and productive for critics.
One particular way to do this would be to talk with people earlier in the process. The discussion I've seen around previewing criticism implicitly assumes there's a full polished blog post, ready to go except for the criticized org's response. That's a very annoying time to find entirely new facts, and imposes a very high tax on criticism. It also makes it easy for a defensive or merely human org to create a bunch more work for the critic without adding value.
If orgs made themselves available earlier in the process, to discuss concerns before someone invested tens of hours doing their own research and carefully writing it up, I would feel much better about strong norms of checking in with them ahead of time. Alas this probably necessitates spending a bunch of time with potential critics whose criticisms are stupid or who were never going to write them, which is pretty costly. But so is spending 30 hours writing a delicate, polished piece of criticism only to have the org create a bunch of work for you.
In this post the criticizer gave the criticizee an opportunity to reply in-line in the published post—in effect, the criticizee was offered the last word. I thought that was super classy, and I’m proud to have stolen that idea on two occasions (1,2).
If anyone’s interested, the relevant part of my email was:
(In the LW/EAF post editor, the inline reply-boxes are secretly just 1×1 tables.)
Another super classy move was I wrote a criticism post once, and the person I criticized retweeted it. (Without even dunking on it!) (The classy person here was Robin Hanson.) I’m proud to say that I’ve stolen that one too, although I guess not every time.
Appendix 3. Other assumptions for the post
Following up on this note: below is a non-exhaustive list of other potentially-relevant assumptions that I won’t bother discussing but that I might rely on (to different degrees) in this post:
(There's almost certainly more.)
Appendix 2. Where I personally think the norm should be on whether to run criticism past people
I think it’s generally good to give people a heads-up if you’re writing a post criticizing their work (and I tentatively think a week or two is a reasonable timeline).
I don’t think we need to push people to be willing to update drafts if they get a private response to their draft before the criticism is posted publicly. I know of multiple cases where the critic got pulled into an extended back-and-forth as a result of doing something like this, and I suspect that this kind of thing sometimes causes people to just not post anything in cases where something should have been posted. (There are still clear benefits to being willing to update drafts, but there are also real costs.)
I also think that running criticism past the people whose work is being criticized should be a soft norm that you can override if your criticism is time-sensitive (e.g. there’s something happening tomorrow that you want to inform the Forum audience about) or if you have good reason to believe that people don’t share your fundamental values, are not acting in good faith, might pressure you to stop you from publishing, or would react in a hostile way for one reason or another. (E.g. if someone were to write a criticism about how MacDonalds shouldn’t be serving meat, I don’t think they should feel obliged to first get in touch with MacDonalds. Jeff notes another example in his “EDIT” here.)
In particular, from a moderation perspective: we will not ban people from the Forum for posting criticism without running it past anyone.
Note: if you’re thinking of sharing very personal information or something closer to an unverified rumor, I’d more strongly push you to run it past people in advance. You can check with the moderation team if you’re not sure whether something is personal. See also the policies on revealing personal information on the Forum.
Appendix 1.[1] Reasons to run criticism past the people whose work you’re criticizing, and reasons to not do it
Please feel free to suggest more!
You can see my overall take here.
This post has a somewhat similar discussion.
These are pretty rough notes that I pulled out to make the main post easier to use as a guide. The comment format is somewhat awkward, but it might work — I'm curious to hear how people feel about it.
Thanks for the pretty useful tips, Lizka!
The email of the person can usually be inferred from the name, and domain of the website of the organisation. For example, to contact you (Lizka Vaintrob) who work at CEA (whose website domain is centreforeffectivealtruism.org), I would consider the following (in this order):
Instead of sending my email to all of the above, I would use Email ID Checker to find the 1st valid one. It looks like the 1st is valid:
“ Are you up for modifying your draft based on private responses from the people whose work you're criticizing?”
If you’ve solicited feedback, I would suggest that your obligation to modify your draft depends on how strong their response is.
If it’s clear that your draft contains significant factual inaccuracies, then publishing reflects poorly on you (though you shouldn’t automatically feel a need to respond to attempts to dispute a fact, as often they may be pointing out something minor or you may have reason to doubt their account).
If the average reader after reading both your post and their response will come away with an impression that your critique was ideological or one-sided, then you should probably edit your draft, though there might be exceptions (say if someone is traumatised to the point where they lack the ability to be objective, so the alternative would be not publishing at all).
To be clear, I’m not saying that you need to edit just because they’ve made some good or reasonable points. I’m more suggesting that you shouldn’t ignore points if this would make a good faith reader feel like your article was pushing them away from the truth.