The illusion of science in comparative cognition

by gavintaylor 1 min read2nd Nov 20193 comments


This is a linkpost for

This article would be worth considering by EAs using animal cognition studies to assess their sentience; it suggests a lot of the literature is biased towards overly positive conclusions about animal cognition.

A prominent vein of comparative cognition research asks which cognitive abilities may be ascribed to different species. Here, we argue that the current structure of comparative cognition makes it near impossible to evaluate the accuracy of many of the claims produced by the field’s empirical research. ... This argument is based on the following observations:
1) Phenomenon-based comparative cognition uses confirmatory research methods that are directionally biased
2) In combination with a publication bias and a likely high rate of false discoveries, this bias suggests our literature contains many false positive findings
3) This directional bias persists even with strong methodological criticism, and when researchers explicitly consider alternative explanations for the phenomena studied
4) No formal method exists for generating and assessing theory-disconfirming evidence that could counter the biased positive evidence
5) Ambiguity in definitions allow us as researchers to flexibly adjust our substantive claims depending on whether we are refuting criticism or selling the results
6) The small size of comparative cognition as a research field perpetuates and reinforces points 1 to 5.

My own research training is in neuroethology, where Krogh's Principle is commonly used as a guide:

for such a large number of problems there will be some animal of choice, or a few such animals, on which it can be most conveniently studied.

While these days the principle is helpful for encouraging people to work on animals other than model organisms, it probably reinforces a lot of the biases above (particularly #1). For instance, desert ants have been studied as a model of visual navigation for many years, but were originally chosen for this because they lived in an environment that was hard to navigate and so initial studies mostly confirmed researchers opinions of their exceptional navigation abilities. If cognition is usually tested in exceptional cases then it probably also reduces the generality of findings between species - most ant species don't live in the Saharan Desert and will have had less selection pressure to evolve exceptional navigation abilities.