PIBBSS Summer Research Fellowship -- Q&A event
Somewhat related: What to do with people? https://forum.effectivealtruism.org/posts/oNY76m8DDWFiLo7nH/what-to-do-with-people
Context: (1) Motivations for fostering EA-relevant interdisciplinary research; (2) "domain scanning" and "epistemic translation" as a way of thinking about interdisciplinary research
The following list of fields and leading questions could be interesting for interdisciplinry AI alignment reserach. I started to compile this list to provide some anchorage for evaluating the value of interdiscplinary research for EA causes, specifically AI alignment.
Some comments on the list:
Very interested in hearing thoughts on the below!
Target domain: AI alignment/safety/governance
Below, I briefly discuss some motivating reasons, as I see them, to foster more interdisciplinary thought in EA. This includes ways EA's current set of research topics might have emerged for suboptimal reasons.
The ocean of knowledge is vast. But the knowledge commonly referenced within EA and longtermism represents only a tiny fraction of this ocean.
I argue that EA's knowledge tradition is skewed for reasons including but not-limited-to the epistemic merit of those bodies of knowledge. There are good reasons for EA to focus in certain areas:
However, some of the reasons we’ve ended up with our current foci may not be as good:
Having a skewed knowledge basis is problematic. There is a significant likelihood that we are missing out on insights or perspectives that might critically advance our undertaking. We don’t know what we don’t know. We have all the reasons to expect that we have blindspots.
I am interested in the potential value and challenges of interdisciplinary research.
(Academic) incentives make it harder for transdisciplinary thought to flourish, resulting in what I expect to be an undersupply thereof. One way of thinking about why we would see an undersupply of interdisciplinry thought is in terms of "market inefficiencies". For one, individual actors are incentivised (because it’s less risky) to work on topics that are already recognised as interesting by the community (“exploitation”), as opposed to venturing into new bodies of knowledge that might or might not prove insightful (“exploration”). What is “already recognized as valuable by the community”, however, will only in part be determined by epistemic considerations, and in another part be shaped by path-dependencies. For two, “markets” are insufficiently liquid and thus tend to fail where we cannot easily specify what we want. I’d argue that this is the case for DS/ET work. This is generally true for intellectual work, but is likely even more true for DS/ET work due to the relatively siloed structure of academia that adds additional “transaction costs” to attempts of communicating across disciplinary boundaries.
One way to reduce these inefficiencies is by improving the interfaces between the disciplines. "Domain scanning" and "episetmic translation" are precisely about creating such interfaces. Their purpose is to identify knowledge that is concretely relevant to a given target domain and make that knolwege accessible to thinkers entrenched in the "vocabulary" of that target domain. A useful interface between political philosophy and computer science, for example, might require a mathematical formalization of central ideas such as justice.
At the same time, doing interdisciplinary well is callenging. For example, interdisciplinary research can only be as valuable as a researcher's ability to identify knowledge relevant to their target domain; or as a research community's quality assurance/error correction mechanisms. Phenomena like citogenesis or motivatiogensis are examples of manifestations of these difficulties. There have been various attempts at overcoming these incentive barriers, for example the Santa Fe Institute whose organizational structure completely disregards scientific disciplines; -ARPAs have a similar flavour; the field of cybernetics which proposed an inherently transdisciplinary view on regulatory systems; or the recent surge in the literature on “mental models” (e.g. here or here).
A closer inspection of such examples - in how far they were successful and how they went about it - might bear some interesting insights. I don't have the capacity to properly puruse such case studies in the near future, but it's definteily something on my list of potentially promising (side) projects.
If readers are aware of other examples of innovative approaches trying to solve this problem that might make for insightful case studies, I’d love to hear them.
The below provides definitions and explanations of "domain scanning" and "epistemic translation", in an attempt of adding further gears to how interdisciplinary research works.
I suggest understanding domain scanning and epistemic translation as a specific type of research that both plays (or ought to play) an important role as part of a larger research progress, or can be usefully pursued as “its own thing”.
By domain scanning, I mean the activity of searching through diverse bodies and traditions of knowledge with the goal of identifying insights, ontologies or methods relevant to another body of knowledge or to a research question (e.g. AI alignment, Longtermism, EA). I call source domains those bodies of knowledge where insights are being drawn from. The body of knowledge that we are trying to inform through this approach is called the target domain. A target domain can be as broad as an entire field or subfield or a specific research problem (in which case I often use the term target problem instead of target domain).
Domain scanning isn’t about comprehensively surveying the entire ocean of knowledge, but instead about selectively scouting for “bright spots” - domains that might importantly inform the target domain or problem.
An important rationale for domain scanning is the belief that model selection is a critical part of the research process. By model selection, I mean the way we choose to conceptualize a problem at a high-level of abstraction (as opposed to, say, working out the details given a certain model choice). In practice, however, this step often doesn’t happen at all because most research happens within a paradigm that is already “in the water”.
As an example, say an economist wants to think about a research question related to economic growth. They will think about how to model economic growth and will make choices according to the shape of their research problem. They might for example decide between using an endogenous growth or an exogenous growth model, and other modeling choices at a similar level of abstraction. However, those choices happen within an already comparably limited space of assumptions - in this case namely neoclassical economics. It's at this higher level of abstraction that I think we're often not sufficiently looking beyond a given paradigm. Like fish in the water.
Neoclassical economics, as an example, is based on assumptions such as agents being rational and homogenous, and the economy being an equilibrium system. Those are, in fact, not straightforward assumptions to make, as heterodox economics have in recent years slowly been bringing to the attention of the field. Complexity economics, for example, drops the above-mentioned assumptions which helps broaden our understanding of economics in ways I think are really important. Notably, complexity economics is inspired by the study of non-equilibrium systems from physics and its conception of heterogeneous and boundedly rational agents come from fields such as psychology and organizational studies.
Research within established paradigms is extremely useful a lot of the time and I am not suggesting that an economist who tackles their research question from a neoclassical angle is necessarily doing something wrong. However, this type of research can only ever make incremental progress. As a research community, I do think we have a strong interest in fostering, at a structural level, the quality of interdisciplinary transfer.
The role of model selection is particularly important in the case of pre-paradigmatic fields (examples include AI Safety or Complexity Science). In this case, your willingness to test different frameworks for conceiving of a given problem seems particularly valuable in expectation. Converging too early on one specific way of framing the problem risks locking in the burgeoning field too early. Pre-paradigmatic fields can often appear fairly chaotic, unorganized and unprincipled (“high entropy”). While this is sometimes evidence against the epistemic merit of a research community, I tend to want to abstain from holding this against emerging fields, because, since the variance of outcomes is higher, the potential upsides are higher too. (Of course, one’s overall judgement of the promise of an emerging paradigm will also depend more than just this factor.)
By epistemic translation, I mean the activity of rendering knowledge commensurable between different domains. In other words, epistemic translation refers to the intellectual work necessary to i) understand a body of knowledge, ii) identify its relevance for your target domain/problem, and iii) render relevant conceptual insights accessible to (the research community of) the target domain, often by integrating it.
Epistemic translation isn’t just about translating one vocabulary into another or merely sharing factual information. It’s about expanding the concept space of the target domain by integrating new conceptual insights and perspectives.
The world is complicated and we are at any one time working with fairly simple models of reality. By analogy, when I look at a three-dimensional cube, I can only see a part of the entire cube at any one time. By taking different perspectives on the same cube and putting these perspectives together - an exercise one might call “triangulating reality” -, I can start to develop an increasingly accurate understanding of the cube. The box inversion hypothesis by Jan Kulveit is another, AI alignment specific example of what I’m thinking about.
I think something like this is true for understanding reality at large, - be it magnitudes more difficult than the cube example suggests. Domain scanning is about seeking new perspectives on your object of inquiry, and epistemic translation is required for integrating these numerous perspectives with one another in an epistemically faithful manner.
In the case of translation between technical and non-technical fields - say translating central notions of political philosophy into game theoretic or CS language - the major obstacle to epistemic translation is formalization. A computer scientist might well be aware of, say, the depth of discourse on topics like justice or democracy. But that doesn’t yet mean that they can integrate this knowledge into their own research or engineering. Formalization is central to creating useful disciplinary interfaces and close to no resources are spent to systematically spreading up this process.
Somewhere in between domain scanning and epistemic translation, we could talk about “prospecting” as the activity of providing epistemic updates on how valuable a certain source domain is likely to be. This involves some scanning and some translation work (therefore categorized as “in between the two”), and would serve the central function of a community mechanism for coordinating around what a community might want to pay attention to.
I think the "so that they become more predictable [to the recommender algorithm]" is crucial in Russel's argument. IF human preferences were malleable in this way, and IF recommender algorithms are strong enough to detect that malleability, then the pressures towards the behaviour that Russel suggests is strong and we have a lot of reasons to expect it. I think the answer to both IFs is likely to be yes.
I agree something about influence is important. As a counterpoint, I think many manifestations of "having influence" don't store well (e.g. the fact that at a given time, a relatively large number of EAs have an "influential role" (whatever that means exactly) is only weakly related to how many EAs will have an influential role in t+1 (say a generation later). Wrt accumulation, influence also seems less straightforward to grow when you compare it to e.g. money (and to a lesser extent to knowledge) which, thanks to interest rates, accumulates at a certain rate basically for free (without you having to do anything) and fairly robustly. I'm not saying that influence is clearly a worse investment than money when it comes to future impact potenital, but that money is a pretty good and stable baseline that might not be as easy to beat as one might think at first sight. Also I think approaches of using "influence" to store and accumulate impact potential will vary a lot on these dimensions, so we'd probably want to talk about such approaches in the concrete rather than the abstract> under your framework, community building is also an intervention for patient longtermism+1 and also worth flagging that e.g. Philip Trammel explicitly says so too in his work on patient longtermism (though he clarifies that this is only true for specific types of community building)
This made me think of the way David Deutsch talks about knoweldge creation - where knowledge manifests physically in e.g. the way a species is adapted to its niche. The process of natural selection that lead to this adapation is a process of "exploratiin" and "error correction" that accumulates knoweldge. That degree of adaptation is the physical manifestation of kowledge. DNA is an important substrate of this process - however, I expect that DNA won't be the most fruitful level of abstraction at which to think about the patient longtermist question.
Still, to explore this framework a bit more ...
I largely agree with your assessment that and how automation puts a lot of pressure on the fate of democracy (although, as you acknowledge, there are ways automation could strengthen democracy, and the way this will cash out sure seems liek it's subject to strong path dependency.)
When we compare pre-industrial times to post-industrial times, it is not only our economy and our arsenal of technologies that is different. Within these ~200-300 years, humanity has also undergone meaningful intellectual and moral progress. This includes things like coming to think that women and people of colour are full members of society, or spelling out values such as freedom, self-realization, etc. If automatisation will lead to power being concentrated in the hands of a small elite, this also means that the beliefs and values of this elite become more important. Of course, if their moral ideals are in stark contrast with others, e.g. economic interests, we should expect they will just throw most of these ideals over board or engage in elaborate rationalizations to present they are still holding them up high. But if the conflict of interst remains relatively weak, I do think this migth be a factor that palys a role.