Effective Altruism Forum
EA Forum

Hide table of contents
Comment Permalink
Allan McCay1
0
0

I meant to add that the first of the papers (2018)  was a response to the work of Yuval Noah Harari in his book Homo Deus - in this Journal of Ethics and Emerging Technologies paper. See here:

The Value of Consciousness and Free Will in a Technological Dystopia 

Reply
See in context
by Allan McCay
2 min read 2

3

AI safetyPhilosophyAI alignment
Frontpage

By Allan McCay

Back in 2018, I began writing about a problem that may be fundamental to AI — one that still receives some but relatively little attention: the problem of incommensurability.

Incommensurability arises when choices involve "apples and oranges" — values or considerations that are so different in kind that they cannot be meaningfully compared by a single metric. It's a problem that philosophers and legal theorists have wrestled with for a long time. And yet it’s now taking center stage as we build increasingly agentic AI systems and edge ever closer to Artificial General Intelligence (AGI).

If we want to align powerful AI systems with human values — especially in contexts involving radically different types of considerations — incommensurability may be a critical stumbling block.

In my new open access article Apples and oranges: AI’s incommensurability problem—  (AI & Society, 2025) — I argue that incommensurability is not just a philosophical curiosity. It has urgent implications for AI alignment, superintelligence, and the future of work.

Three Key Claims

  1. Less Automation?
    Incommensurability may limit the kinds of cognitive tasks AI can do well. In domains like legal work, where decisions often involve apples-and-oranges trade-offs (justice vs efficiency, rights vs risk), this could slow down full automation — though not stop it entirely. See also:
  2. Superishintelligence, Not Superintelligence
    A system that excels at certain cognitive tasks but fails to rationally resolve incommensurable value conflicts may only ever be “super-ish” — not truly superintelligent. This cognitive Achilles’ heel could reshape how we think about AI’s limits.
  3. Alignment Risk
    If we can’t solve incommensurability then alignment itself becomes even trickier. How do we align an agent that must choose between outcomes it can’t compare on a shared scale?

So What Can Be Done?

In the article, I offer several concrete suggestions for developers and AI safety researchers. 

We might need models that recognize when they are in apples-and-oranges territory, and mechanisms to address such decisions — especially in high-stakes environments.

We also need to think of ways of assessing how models are performing on incommensurability, perhaps by way of a sort of Turing Test of incommensurability.

Who Should Read This?

This post — and the article it introduces — is especially relevant for:

  • Researchers working on frontier AI models
  • Teams developing AI safety strategies and alignment frameworks
  • Philosophers, lawyers, and interdisciplinary thinkers concerned with value pluralism
  • Effective altruists thinking about long-term AI risk and how best to allocate research effort

Final Thought

In the race toward AGI, it's tempting to focus only on scale, compute, and benchmarks. But maybe we also need to pause and ask: what happens when our systems face choices that resist comparison — when we give them apples and oranges?

If we don’t reckon with incommensurability now, we may end up with superishintelligence that’s misaligned in ways we never anticipated.

Feel free to read and share the full article here (Open Access).
I'd welcome feedback, collaboration, and critical engagement — especially from those working on AI safety, alignment, and policy.

3

0
1

Reactions

0
1

More posts like this

Comments2


Sorted by
New & upvoted
Click to highlight new comments since:
Noah Birnbaum1
0
0

From a philosophy standpoint, I find incommensurability pretty implausible (at least to act upon) for a couple reasons: 

  1. If two values are incommensurable, for every action that you take, there is some probability that you are making a trade-off between these actions. Given some version of expected value to be correct (where some probability of value is equivalent to trading off some lower value itself), this would mean that every action one takes is making a choice between two incommensurable goods. This seems to lock you into a constant state of decision paralysis (where every action you take is trading off two incommensurable goods), which, I believe, should just make incommensurable goods a non-viable option. (See this paper for more discussion)
  2. Imagine you have some credence in two things being incommensurable (thereby making it such that you have no reasons to act in either way). Even if this is the case, however, you should still have some non-zero credence in these values / actions being commensurable as it is a contingent proposition. If the credence of incommensurability gives you no reason to act and the credence of commensurability does give you reason to act, this makes incommensurability irrelevant, making it so that your actions should entirely be informed by the case conditional on commensurability. 

Happy to chat more about this, if you think that you'd find that helpful. 

Reply
Allan McCay1
0
0

I meant to add that the first of the papers (2018)  was a response to the work of Yuval Noah Harari in his book Homo Deus - in this Journal of Ethics and Emerging Technologies paper. See here:

The Value of Consciousness and Free Will in a Technological Dystopia 

Reply
Curated and popular this week
 ·  · 5m read
 · 
[Cross-posted from my Substack here] If you spend time with people trying to change the world, you’ll come to an interesting conundrum: Various advocacy groups reference previous successful social movements as to why their chosen strategy is the most important one. Yet, these groups often follow wildly different strategies from each other to achieve social change. So, which one of them is right? The answer is all of them and none of them. This is because many people use research and historical movements to justify their pre-existing beliefs about how social change happens. Simply, you can find a case study to fit most plausible theories of how social change happens. For example, the groups might say: * Repeated nonviolent disruption is the key to social change, citing the Freedom Riders from the civil rights Movement or Act Up! from the gay rights movement. * Technological progress is what drives improvements in the human condition if you consider the development of the contraceptive pill funded by Katharine McCormick. * Organising and base-building is how change happens, as inspired by Ella Baker, the NAACP or Cesar Chavez from the United Workers Movement. * Insider advocacy is the real secret of social movements – look no further than how influential the Leadership Conference on Civil Rights was in passing the Civil Rights Acts of 1960 & 1964. * Democratic participation is the backbone of social change – just look at how Ireland lifted a ban on abortion via a Citizen’s Assembly. * And so on… To paint this picture, we can see this in action below: Source: Just Stop Oil which focuses on…civil resistance and disruption Source: The Civic Power Fund which focuses on… local organising What do we take away from all this? In my mind, a few key things: 1. Many different approaches have worked in changing the world so we should be humble and not assume we are doing The Most Important Thing 2. The case studies we focus on are likely confirmation bias, where
 ·  · 2m read
 · 
I speak to many entrepreneurial people trying to do a large amount of good by starting a nonprofit organisation. I think this is often an error for four main reasons. 1. Scalability 2. Capital counterfactuals 3. Standards 4. Learning potential 5. Earning to give potential These arguments are most applicable to starting high-growth organisations, such as startups.[1] Scalability There is a lot of capital available for startups, and established mechanisms exist to continue raising funds if the ROI appears high. It seems extremely difficult to operate a nonprofit with a budget of more than $30M per year (e.g., with approximately 150 people), but this is not particularly unusual for for-profit organisations. Capital Counterfactuals I generally believe that value-aligned funders are spending their money reasonably well, while for-profit investors are spending theirs extremely poorly (on altruistic grounds). If you can redirect that funding towards high-altruism value work, you could potentially create a much larger delta between your use of funding and the counterfactual of someone else receiving those funds. You also won’t be reliant on constantly convincing donors to give you money, once you’re generating revenue. Standards Nonprofits have significantly weaker feedback mechanisms compared to for-profits. They are often difficult to evaluate and lack a natural kill function. Few people are going to complain that you provided bad service when it didn’t cost them anything. Most nonprofits are not very ambitious, despite having large moral ambitions. It’s challenging to find talented people willing to accept a substantial pay cut to work with you. For-profits are considerably more likely to create something that people actually want. Learning Potential Most people should be trying to put themselves in a better position to do useful work later on. People often report learning a great deal from working at high-growth companies, building interesting connection
 ·  · 31m read
 · 
James Özden and Sam Glover at Social Change Lab wrote a literature review on protest outcomes[1] as part of a broader investigation[2] on protest effectiveness. The report covers multiple lines of evidence and addresses many relevant questions, but does not say much about the methodological quality of the research. So that's what I'm going to do today. I reviewed the evidence on protest outcomes, focusing only on the highest-quality research, to answer two questions: 1. Do protests work? 2. Are Social Change Lab's conclusions consistent with the highest-quality evidence? Here's what I found: Do protests work? Highly likely (credence: 90%) in certain contexts, although it's unclear how well the results generalize. [More] Are Social Change Lab's conclusions consistent with the highest-quality evidence? Yes—the report's core claims are well-supported, although it overstates the strength of some of the evidence. [More] Cross-posted from my website. Introduction This article serves two purposes: First, it analyzes the evidence on protest outcomes. Second, it critically reviews the Social Change Lab literature review. Social Change Lab is not the only group that has reviewed protest effectiveness. I was able to find four literature reviews: 1. Animal Charity Evaluators (2018), Protest Intervention Report. 2. Orazani et al. (2021), Social movement strategy (nonviolent vs. violent) and the garnering of third-party support: A meta-analysis. 3. Social Change Lab – Ozden & Glover (2022), Literature Review: Protest Outcomes. 4. Shuman et al. (2024), When Are Social Protests Effective? The Animal Charity Evaluators review did not include many studies, and did not cite any natural experiments (only one had been published as of 2018). Orazani et al. (2021)[3] is a nice meta-analysis—it finds that when you show people news articles about nonviolent protests, they are more likely to express support for the protesters' cause. But what people say in a lab setting mig
Recent opportunities in AI safety
121
AI Moral Alignment: The Most Important Goal of Our Generation
Ronen Bar
· · 10m read
31
31
24
Last week to apply for the Futurekind AI Fellowship! (deadline: April 1)
Jay Luong
· · 1m read
0
0
21
Developing AI Safety: Bridging the Power-Ethics Gap (Introducing New Concepts)
Ronen Bar
· · 6m read
3
3
View more