Research Deprioritizing External Communication

Jeff Kaufman 🔸

Comments 26

Sorted by

New & upvoted

Partly unrelated: at first, I thought the title meant that we should research deprioritizing external communication. It took me a while to understand it meant that research is/was deprioritizing external communication.

Emrik

Edit: The post has excellent nuance, and I make no claim to support or defend Leverage specifically (idk them). My comment is intended more generally, and my disagreement concerns two points:

"The core problem was that Leverage 1.0 quickly became much too internally focused."
"If they're not publishing research I would take that as a strong negative signal for the project."

You make several points, but I just want to respond to my impression that you're trying to anchor wayward researchers or research groups to the "main paradigm" to decrease the chance that they'll be wrong. I'm pretty strongly against this.

In a common-payoff game (like EA research), we all share the fruits of major discoveries regardless of who makes the discovery. So we should heavily prioritise sensitivity over specificity. It doesn't matter how many research groups are wildly wrong, as long at least one research group figures out how to build the AI that satisfies our values with friendship and ponies. So when you're trying to rein in researchers instead of letting them go off and explore highly variable crazy stuff, you're putting all your eggs in one basket (the most respectable paradigm). Researchers are already heavily incentivised to research what other people are researching (the better to have a lively conversation!), so we do not need additional incentives against exploration.

The value distribution of research fruits is fat-tailed (citation needed). Strategies that are optimal for sampling normal distributions are unlikely to be optimal for fat tails. Sampling for outliers means that you should rely more on theoretical arguments, variability, and exploration, because you can't get good data on the outliers--the only data that matters. If you insist on being legible and scientific, so you optimise your strategy based on the empirical data you can collect, you're being fooled into mediocristan again.

Lemme cite a paper in network epistemology so I can fake looking like I know what I'm talking about,

“However, pure populations of mavericks, who try to avoid research approaches that have already been taken, vastly outperform the other strategies. Finally, we show that, in mixed populations, mavericks stimulate followers to greater levels of epistemic production, making polymorphic populations of mavericks and followers ideal in many research domains.”^[1]
-- Epistemic landscapes and the division of cognitive labor

That said, I also advocate against explorers being allowed to say

But I'm virtuously doing high-variance exploration, so I don't need to worry about your rigorous schmigorous epistemology!

Explorers need to be way more epistemologically vigilant than staple researchers pursuing the safety of existing paradigms. If you leave your harbour to sail out into the open waters, that's not a good time to forget your sextant, or pretend you'll be a better navigator without studying the maps that do exist.

^{^}
FWIW, I think conclusions from network-epistemological computer simulations are extremely weak evidence about what we as an irl research community should do, and I mainly benefit from it because they occasionally reveals patterns that help with analysing real-life phenomena. The field exists at all--despite their obviously irrelevant "experiments"--because it makes theoretical speculation seem more technical, impressive, professional.

Jeff Kaufman 🔸

It doesn't matter how many research groups are wildly wrong, as long at least one research group figures out how to build the AI that satisfies our values with friendship and ponies.

Sort of? In your hypothetical there are two ways your research project could go once you believe you've succeeded:

You go and implement it, or
You figure out how to communicate your results to the rest of the industry.

If you go with (1) then it's really important that you get things right, and if you've disconnected yourself from external evaluation I think there's a large chance you haven't. I'd much prefer to see (2), except now you do need to communicate your results in detail so the rest of the world can evaluate and so you didn't gain that much by putting off the communication until the end.

I'll also make a stronger claim, which is that communication improves your research and chances of success: figuring out how to communicate things to people who don't have your shared context makes it a lot clearer which things you actually don't understand yet.

trying to rein in researchers instead of letting them go off and explore highly variable crazy stuff

I'm not sure why you think I'm advocating avoiding high-variability lines of research? I'm saying research groups should make public updates on their progress to stay grounded, not that they should only take low-risk bets.

Emrik

I edited my original comment to point out my specific disagreements. I'm now going to say a selection of plausibly false-but-interesting things, and there's much more nuance here that I won't explicitly cover because that'd take too long. It's definitely going to seem very wrong at first glance without the nuance that communicates the intended domain.

I feel like I'm in a somewhat similar situation to Leverage, only in the sense that I feel like having to frequently publish would hinder my effectiveness. It would make it easier for others to see the value of my work, but in my own estimation that trades off against maximising actual value.

This isn't generally the case for most research, and I might be delusional (ime 10%) to think it's the case for my own, but I should be following the gradient of what I expect will be the most usefwl. It would be selfish of me to do the legible thing motivated just by my wish for people to respect me.

The thing I'm arguing for is not that people like me shouldn't publish at all, it's that we should be very reluctant to punish gambling sailors for a shortage of signals. They'll get our attention once they can demonstrate their product.

The thing about having to frequently communicate your results is that it incentivises you to adopt research strategies that lets you publish frequently. This usually means forward-chaining to incremental progress without much strategic guidance. Plus, if you get into the habit of spending your intrinsic motivation on distilling your progress to the community, now your brain's shifted to searching for ideas that fit into the community, instead of aiming your search to solve the highest-priority confusion points in your own head.

To be an effective explorer, you have to get to the point where you can start to iterate on top of your own ideas. If you timidly "check in" with the community every time you think you have a novel thought, before you let yourself stand on it in order to explore further down the branch, then 1) you're wasting their time, and 2) no one's ever gonna stray far from home.

When you go from—

A) "huh, I wonder how this thing works, and how it fits into other things I have models of."
to
B) "hmm, the community seems to behave as if Y is true, but I have a suspicion that ¬X,
so I should research it and provide them with information they find valuable."

—then a pattern for generating thoughts will mostly be rewarded based on your prediction about whether the community is likely to be persuaded by those thoughts. This makes it hard to have intrinsic motivation to explore anything that doesn't immediately seem relevant to the community.

And while B is still reasonably aligned with producing value as long as the community is roughly as good at evaluating the claims as you are, it breaks down for researchers who are much better than their expected audience at what they specialise in. If the most competent researchers have brains that optimise for communal persuasiveness, they're wasting their potential when they could be searching for ideas that optimise for persuading themselves--a much harder criteria to meet given that they're more competent.

I think it's unhealthy to–within your own brain–constantly try to "advance the communal frontier". Sure, that could ultimately be the goal, but if you're greedily and myopically only able to optimise for specifically that at every step, then that is like a chess player who's compulsively only able to look for checkmate patterns–unable to see forks that merely win material or positional advantage.

How frequently do you have to make your progress legible to measurable or consensus criteria? How lenient is your legibility loop?

I'm not saying it's easy to even start trying to feel intrinsic motivation for building models in your own mind based on your own criteria for success, but being stuck in a short legibility loop certainly doesn't help.

If you've learned to play an instrument, or studied painting under a mentor, you may have heard the advice "you need to learn you trust in your own sense of aesthetics." Think of the kid who, while learning the piano, expectantly looks to their parent after every key they press. They're not learning to listen. Sort of like a GAN with a discriminator trusted so infrequently that it never learns anything. Training to both generate and discriminate within yourself, using your own observations, will be pretty embarrassing at first, but you're running a much shorter feedback loop.

Jeff Kaufman 🔸

Maybe we're talking about different timescales here? I definitely think researchers need to be able to make progress without checking in with the community at every step, and most people won't do well to try and publish their progress to a broad group, say, weekly. For a typical researcher in an area with poor natural feedback loops I'd guess the right frequency is something like:

Weekly: high-context peers (internal colleagues / advisor / manager)
Quarterly: medium-context peers (distant internal colleagues / close external colleagues)
Yearly: low-context peers and the general world

(I think there are a lot of advantages to writing for these, including being able to go back later, though there are also big advantages to verbal interaction and discussion.)

I think Leverage was primarily short on (3); from the outside I don't know how much of (2) they were doing and I have the impression they were investing heavily in (1).

Emrik

Roughly agreed. Although I'd want to distinguish between feedback and legibility-requirement loops. One is optimised for making research progress, the other is optimised for being paid and respected.

When you're talking to your weekly colleagues, you have enough shared context and trust that you can ramble about your incomplete intuitions and say "oops, hang on" multiple times in an exposition. And medium-context peers are essential for sanity-checking. This is more about actually usefwl feedback than about paying a tax on speed to keep yourself legible to low-context funders.

Thank you for chatting with me! ^^

Jeff Kaufman 🔸

(I'm only trying to talk about feedback here as it relates to research progress, not funding etc.)

Emrik

Ah, but part of my point is that they're inextricably linked--at least for pre-paradigmatic research that requires creativity and don't have cheap empirical-legible measures of progress. Shorter legibility loops puts a heavy tax on the speed of progress, at least for the top of the competence distribution. I can't make very general claims here given how different research fields and groups are, but I don't want us to be blind to important considerations.

There are deeper models behind this claim, but one point is that the "legibility loops" you have to obey to receive funding requires you to compromise between optimisation criteria, and there are steeper invisible costs there than people realise.

iporphyry

Edit: I mostly retract this comment. I skimmed and didn't read the post carefully (something one should never do before leaving a negative comment) and interpreted it as "Leverage wasn't perfect, but it is worth trying to make Leverage 2.0 work or have similar projects with small changes". On rereading, I see that Jeff's emphasis is more on analyzing and quantifying the failure modes than on salvaging the idea.

That said, I just want to point out that (at least as far as I understand it), there is a significant collection of people within and around EA who think that Leverage is a uniquely awful organization which suffered a multilevel failure extremely reminiscent of your run-of-the mill cult (not just for those who left it, but also for many people who are still in it), which soft-core threatens members to avoid negative publicity, exerts psychological control on members in ways that seem scary and evil. This is context that I think some people reading the sterilized publicity around Leverage will lack.

~~There are many directions from which people could approach Leverage 1.0, but the one that I'm most interested in is lessons for people considering attempting similar things in the future.~~

~~I think there's a really clear lesson here: don't.~~

I'll elaborate: Leverage was a multilevel failure. A fundamentally dishonest and charismatic leader. A group of people very convinced that their particular chain of flimsy inferences led them to some higher truth that gave them advantages over everyone else. A frenzied sense of secrecy and importance. Ultimately, psychological harm and abuse.

It is very clearly a negative example, and if someone is genuinely trying to gain some positive insight into a project from "things they did right" (or noticeably imitate techniques from that project), that would make me significantly less likely to think of them as being on the right track.

There are examples of better "secret projects" - the Manhattan project as well as other high-security government organizations, various secret revolutionary groups like the early US revolutionaries, the abolitionist movement and the underground railroad, even various pro-social masonic orders. Having as one's go-to example of something to emulate an organization that significantly crossed the line into cult territory (or at least into Aleister Crowley level grandiosity around a bad actor) would indicate to me a potential enlarged sense of self-importance, an emphasis on deference and exclusivity ("being on our team") instead of competence and accountability, and a lack of emphasis on appropriate levels of humility and self-regulation.

To be clear, I believe in decoupling and don't think it's wrong to learn from bad actors. But with such a deeply rotten track record, and so many decent organizations that are better than it along all parameters, Leverage is perhaps the clearest example of a situation where people should just "~~say oops" and stop looking for ways to gain any value from it (other than as a cautionary tale) that I have heard of in the EA/LW community.~~

Jeff Kaufman 🔸

attempting similar things in the future

I intended this a bit more broadly than you seem to have interpreted it; I'm trying to include exploratory research groups in general.

gain any value from it (other than as a cautionary tale)

That is essentially what this post is: looking in detail at one specific way I think things went wrong, and thinking about how to avoid this in the future.

I expect tradeoffs around how much you should prioritize external communication will continue to be a major issue for research groups!

iporphyry

Fair enough. I admit that I skimmed the post quickly, for which I apologize, and part of this was certainly a knee-jerk reaction to even considering Leverage as a serious intellectual project rather than a total failure as such, which is not entirely fair. But I think maybe a version of this post I would significantly prefer would first explain your interest in Leverage specifically: that while they are a particularly egregious failure of the closed-research genre, it's interesting to understand exactly how they failed and how the idea of a fast, less-than-fully transparent think tank can be salvaged. It does bother me that you don't try to look for other examples of organizations that do some part of this more effectively, and I have trouble believing that they don't exist. It reads a bit like an analysis of nation-building that focuses specifically on the mistakes and complexities of North Korea without trying to compare it to other less awful entities.

Jeff Kaufman 🔸

worth trying to make Leverage 2.0 work

Note that Leverage 2.0 is a thing, and seems to be taking a very different approach towards the history of science, with regular public write-ups: https://www.leverageresearch.org/history-of-science

Linch

It seems like you're misreading Jeff's post. Perhaps deliberately. I will prefer it if people on this forum do this less.

[This comment is no longer endorsed by its author]

iporphyry

Certainly not deliberately. I'll try to read it more carefully and update my comment

Linch

Thanks. I've retracted my comment since I think it's too harsh. <3

iporphyry

Thanks! But I see your point

[anonymous]

That said, I just want to point out that (at least as far as I understand it), there is a significant collection of people within and around EA who think that Leverage is a uniquely awful organization which suffered a multilevel failure extremely reminiscent of your run-of-the mill cult (not just for those who left it, but also for many people who are still in it), which soft-core threatens members to avoid negative publicity, exerts psychological control on members in ways that seem scary and evil. This is context that I think some people reading the sterilized publicity around Leverage will lack.

I can’t comment on whether rumors like this still persist in the EA community, but to the degree that they do, I think there is now a substantial amount of available information that allows for a more nuanced picture of the organization and the people involved.

Two of the best, in my view, are Cathleen’s post and our Inquiry Report. Both posts are quite lengthy, but as you seem passionate about this topic, they may nevertheless be worth reading.

I think it’s fair to say that the majority of people involved in Leverage would strongly disagree with your characterization of the organization. As someone who works at Leverage and was friends with many of the people involved previously, I can say that your characterization strongly mismatches my experience.

Rockwell

Meta note: I think it encourages in-group/out-group experiences on the Forum when known individuals are identified only by their first names and at a minimum would like to see e.g. Geoff, Larissa, and Catherine named in full at least once in this post.

Jeff Kaufman 🔸

I've intentionally used only first names for everyone in the post, including for individuals who are not well known, to make this post less likely to show up when searching anyone's name.

LarissaVaughanRowe

I just wanted to say thank you for doing this Jeff. I sympathize with Rockwell Schwartz’s general point, but since Cathleen’s post asks that people not use her full name or name her former colleagues I appreciate you taking this seriously.

(For clarity, I don’t mind people using my full name. It’s my forum username and very easily found e.g. on Leverage’s website. But I currently work at Leverage Research and decided to work there knowing full well how some people in EA react when the topic of Leverage comes up. The same is not true of everyone, and I think individuals who have not chosen to be public figures should be allowed to live in peace should they wish to).

Rockwell

That makes sense and I wasn't familiar with Cathleen's request or the general aims of quasi-anonymity here. I think it is useful to specify that you are intentionally not using full names because otherwise the assumption is likely that these are people one should know and contributes to my above concern.

[anonymous]

Instead I would specifically look at its output and approach to external engagement: if they're not publishing research I would take that as a strong negative signal for the project. Likewise, in participating in a research project I would want to ensure that we were writing publicly and opening our work to engaged and critical feedback.

I'm curious about why your conclusion is about the importance of public engagement instead of about the importance (and difficulty) of setting up good feedback loops for research.

It seems to me that it is possible to have good feedback loops without good public engagement (e.g., the Manhattan Project) and good public engagement without good feedback loops (e.g., many areas of academic research). But, whereas important research progress seems possible in the former case, it seems all but impossible in the latter case.

Jeff Kaufman 🔸

I think feedback loops are the important thing, but public engagement is a powerful way to strengthen them which Leverage seemed to have suffered from deprioritizing.

In the example of the Manhattan Project, they were studying and engineering physical things, which makes it a lot harder to be wrong about whether you're making progress. My understanding is also that they brought a shockingly high fraction of the experts in the field into the project, which might mean you could get some of what you'd normally get from public presentation internally?

[anonymous]

The degree to which public presentation is likely to strengthen your feedback loops seems to depend quite a lot on the state of the field that you are investigating. In highly functional fields like those found in modern physics, it seems quite likely to be helpful. In less functional fields or those with fewer relevant researchers, this seems less helpful.

To my mind, one strong consideration in favor of publicly presenting your research if you're working in a less functional field is that even if you're right, causing future researchers to build on your work is extremely difficult. Indeed, promising research avenues that are presented publicly die all the time (e.g., muscle reading or phlogiston c.f. Chang in Is Water H2O?). Presenting your research publicly is the best way to engage with other researchers and ensure that, if you do succeed, a research tradition can be built on top of your work.

LarissaVaughanRowe

Larissa from Leverage Research here. I think there might be an interesting discussion to be had about the relationship between feedback loops, external communication (engaging with your main external audiences), and public communication (trying to communicate ideas to the wider public).

For a lot of the history of scientific developments, sharing research, let alone widely distributing it was expensive and rare. Early discoveries in the history of electricity, for example, were nonetheless still made, often by researchers who shared little until they had a complete theory, or a new instrument to display. Often the feedback loops were simply direct engagement with the phenomena itself. Only in more recent history has it become cheap and easy enough to widely share research such that this has become the norm. Similarly, as a couple of people have mentioned in the comments, there are more recent examples of groups that have done great research while having little external engagement: Lockheed Martin and the Manhattan Project being two well-known examples.

This suggests that it is feasible to have feedback loops while doing little external communication of any kind. During Leverage 1.0^[1] people relied more on feedback from their own experiences, interactions with teammates’ experiences and views, workshops and coaching.

That said, we do believe (for reasons independent of research feedback loops) that it was a mistake to not do more external communication in the past, which is why this is something Leverage Research has focused on since 2019. More recently, we have also come to think that it is also important to try to communicate to the wider public (in ways that can be broadly understood) as opposed to just your core audience or peer group. One reason for this is that if projects are only communicated about, and criticisms only accepted in, the language of the particular group that developed them, it's easy for blindspots to remain until it is too late. (I recommend Glen Weyl's "Why I'm Not A Technocrat" for a more detailed treatment of this topic.)

For anyone interested in some of our other reflections on public engagement, I recommend reading our 2019-2020 annual report or our Experiences Inquiry Report. The former is Leverage Research's first annual report since the re-organization in 2019, and one topic we discuss is our new focus on external engagement. The latter shares findings from our inquiry last year into the experiences of former collaborators during Leverage 1.0. To see our engagement efforts today, I recommend checking out our website, subscribing to our newsletter, or following us on Twitter or Medium.

For those interested in the exploratory psychology research Jeff mentions, we recommend reading our write-up from earlier this year covering our 2017 - 2019 Intention Research and keeping an eye on our Exploratory Psychology Research Program page. We are currently working on two pieces: one on risks from introspection (we discuss this a bit on Twitter here), and one on Belief Reporting (an introspective tool developed during Leverage 1.0). We're also thinking of sharing a few documents written pre-2019 that relate to introspection techniques. These would perhaps be less accessible for a wider audience unfamiliar with our introspective tools but may nonetheless be of interest to those who want to dive deeper on our introspective research. All of this will be added to our website when completed.

Finally, I just wanted to thank Jeff for engaging with us in a discussion of his post. Although we disagreed on some things and it ended up a lengthy discussion, I do feel like I came to understand a bit more of where the disagreement stemmed from, and the post was improved through the process. This seems valuable, so I would like to see that norm encouraged.

^{^}
As context, "Leverage 1.0" is the somewhat clumsy term I introduced as a shorthand for the decentralized research collaboration between a few organizations from 2011 to 2019 that's commonly referred to as "Leverage," so as to distinguish it from Leverage Research the organization since 2019 which looks very different.

Emrik

Do you have recommendations for the most enlightening concepts, models, or data produced by Leverage that's not been filtered for public sensibilities? I want to efficiently evaluate whether I wish to dive deeper, and if a write-up is wastefwly optimised for professionalism, I take that as a strong signal against the likelihood that there's significant value here.

I'm not interested in whether you produce research that I expect others will expect to be praised for praising me for approving of, I just want to know if it can help me understand stuff.

Comments