Linch
Posts 85
Comments2917
EDIT: I noticed that in my examples I primed Claude a little, and when unprimed Claude does not reliably (or usually) get to the answer. However Claude 4.xs are still noticeable in how little handholding they need for this class of conceptual errors, Geminis often takes like 5 hints where Claude usually gets it with one. And my impression was that Claude 3.xs were kinda hopeless (they often don't get it even with short explanations by me, and when they do, I'm not confident they actually got it vs just wanted to agree).
"Most people make the mistake of generalizing from a single data point. Or at least, I do." - SA
When can you learn a lot from one data point? People, especially stats- or science- brained people, are often confused about this, and frequently give answers that (imo) are the opposite of useful. Eg they say that usually you can’t know much but if you know a lot about the meta-structure of your distribution (eg you’re interested in the mean of a distribution with low variance), sometimes a single data point can be a significant update.
This type of limited conclusion on the face of it looks epistemically humble, but in practice it's the opposite of correct. Single data points aren’t particularly useful when you know a lot, but they’re very useful when you have very little knowledge to begin with. If your uncertainty about a variable in question spans many orders of magnitude, the first observation can often reduce more uncertainty than the next 2-10 observations put together[1]. Put another way, the most useful situations for updating massively from a single data point are when you know very little to begin with.
For example, if an alien sees a human car for the first time, the alien can make massive updates on many different things regarding Earthling society, technology, biology and culture. Similarly, an anthropologist landing on an island of a previously uncontacted tribe can rapidly learn so much about a new culture from a single hour of peaceful interaction [2].
Some other examples:
- Your first day at a new job.
- First time visiting a country/region you previously knew nothing about. One afternoon in Vietnam tells you roughly how much things cost, how traffic works, what the food is like, languages people speak, how people interact with strangers.
- Trying a new fruit for the first time. One bite of durian tells you an enormous amount about whether you'll like durian.
- Your first interaction with someone's kid tells you roughly how old they are, how verbal they are, what they're like temperamentally. You went from "I know nothing about this child" to a working model.
Far from idiosyncratic and unscientific, these forms of "generalizing from a single data point" are just very normal, and very important, parts of normal human life and street epistemology.
This is the point that Douglas Hubbard tries to hammer in repeatedly over the course of his book, How to Measure Anything: You know less than you think you do, and a single measurement can sometimes be a massive update.
[1] this is basically tautological from a high-entropy prior.
[2] I like Monolingual Fieldwork as a demonstration for the possibilities in linguistics: https://www.youtube.com/watch?v=sYpWp7g7XWU&t=2s
The significance, as I read it, is that you can now trust Claude roughly like a reasonable colleague for spotting such mistakes, both in your own drafts and in texts you rely on at work or in life.
I wouldn't go quite this far, at least from my comment. There's a saying in startups, "never outsource your core competency", and unfortunately reading blog posts and spotting conceptual errors of a certain form is a core competency of mine. Nonetheless I'd encourage other Forum users less good at spotting errors (which is most people) to try to do something like this and post posts that seem a little fishy to Claude and see if it's helpful.[1]
For me, Claude is more helpful for identifying factual errors, and for challenging my own blog posts at different levels (eg spelling, readability, conceptual clarity, logical flow, etc). I wouldn't bet on it spotting conceptual/logical errors in my posts I missed, but again, I have a very high opinion of myself here.
- ^
(To be clear I'm not sure the false positives/false negatives ratio is good enough for other people).
Recent generations of Claude seem better at understanding blog posts and making fairly subtle judgment calls than most smart humans. These days when I’d read an article that presumably sounds reasonable to most people but has what seems to me to be a glaring conceptual mistake, I can put it in Claude, ask it to identify the mistake, and more likely than not Claude would land on the same mistake as the one I identified.
I think before Opus 4 this was essentially impossible, Claude 3.xs can sometimes identify small errors but it’s a crapshoot on whether it can identify central mistakes, and certainly not judge it well.
It’s possible I’m wrong about the mistakes here and Claude’s just being sycophantic and identifying which things I’d regard as the central mistake, but if that’s true in some ways it’s even more impressive.
Interestingly, both Gemini and ChatGPT failed at these tasks. (They can sometimes directionally approach the error I identified, but their formulation is imprecise and broad, and they only have it in a longer list of potential quibbles rather than zero in on the most damning issue).
For clarity purposes, here are 3 articles I recently asked Claude to reassess (Claude got the central error in 2/3 of them). I'm also a little curious what the LW baseline is here; I did not include my comments in my prompts to Claude.
https://terrancraft.com/2021/03/21/zvx-the-effects-of-scouting-pillars/
https://www.clearerthinking.org/post/what-can-a-single-data-point-teach-you
https://www.lesswrong.com/posts/vZcXAc6txvJDanQ4F/the-median-researcher-problem-1
The dynamics you discuss here follow pretty intuitively from the basic conflict/mistake paradigm.
I think it's very easy to believe that the natural extension of the conflicts/mistakes paradigm is that policy fights are composed of a linear combination of the two. Schelling's "rudimentary/obvious" idea, for example, that conflict is and cooperation is often structurally inseparable, is a more subtle and powerful reorientation than it first seems.
But this is a hard point to discuss (because it's in the structure of an "unknown known"), and I didn't interview many people before writing my thing.
If you've never felt tempted before to see the policy disagreements this way, then I could just be miscalibrated here on the obviousness or lack thereof of these ideas.
I like Scott's Mistake Theory vs Conflict Theory framing, but I don't think this is a complete model of disagreements about policy, nor do I think the complete models of disagreement will look like more advanced versions of Mistake Theory + Conflict Theory.
To recap, here's my short summaries of the two theories:
Mistake Theory: I disagree with you because one or both of us are wrong about what we want, or how to achieve what we want)
Conflict Theory: I disagree with you because ultimately I want different things from you. The Marxists, who Scott was originally arguing against, will natively see this as about individual or class material interests but this can be smoothly updated to include values and ideological conflict as well.
I polled several people about alternative models for political disagreement at the same level of abstraction of Conflict vs Mistake, and people usually got to "some combination of mistakes and conflicts." To that obvious model, I want to add two other theories (this list is incomplete).
Consider Thomas Schelling's 1960 opening to Strategy of Conflict
The book has had a good reception, and many have cheered me by telling me they liked it or learned from it. But the response that warms me most after twenty years is the late John Strachey’s. John Strachey, whose books I had read in college, had been an outstanding Marxist economist in the 1930s. After the war he had been defense minister in Britain’s Labor Government. Some of us at Harvard’s Center for International Affairs invited him to visit because he was writing a book on disarmament and arms control. When he called on me he exclaimed how much this book had done for his thinking, and as he talked with enthusiasm I tried to guess which of my sophisticated ideas in which chapters had made so much difference to him. It turned out it wasn’t any particular idea in any particular chapter. Until he read this book, he had simply not comprehended that an inherently non-zero-sum conflict could exist. He had known that conflict could coexist with common interest but had thought, or taken for granted, that they were essentially separable, not aspects of an integral structure. A scholar concerned with monopoly capitalism and class struggle, nuclear strategy and alliance politics, working late in his career on arms control and peacemaking, had tumbled, in reading my book, to an idea so rudimentary that I hadn’t even known it wasn’t obvious.
I claim that this "rudimentary/obvious idea," that the conflict/cooperative elements of many human disagreements is structurally inseparable, is central to a secret third thing distinct from Conflict vs Mistake. If you grok the "obvious idea," we can derive something like
Negotiation Theory(?): I have my desires. You have yours. I sometimes want to cooperate with you, and sometimes not. I take actions maximally good for my goals and respect you well enough to assume that you will do the same; however in practice a "hot war" is unlikely to be in either of our best interests.
In the Negotiation Theory framing, disagreement/conflict arises from dividing the goods in non-zerosum games. I think the economists/game theorists' "standard models" of negotiation theory is natively closer to "conflict theory" than "mistake theory." (eg, often their models assume rationality, which means the "can't agree to disagree" theorems apply). So disagreements are due to different interests, rather than different knowledge. But unlike Marxist/naive conflict theory, we see that conflicts are far from desired or inevitable, and usually there are better trade deals from both parties' lights than not coordinating, or war.
(Failures from Negotiation Theory's perspectives often centrally look like coordination failures, though the theory is broader than that and includes not being able to make peace with adversaries)/
Another framing that is in some ways a synthesis and in some ways a different view altogether that can be simultaneously true with any of the previous theories:
Motivated Cognition: People disagree because their interests shape their beliefs. Political disagreements happen because one or both parties are mistaken about the facts, and those mistakes are downstream of material or ideological interests shading one's biases. Upton Sinclair: “It is difficult to get a man to understand something, when his salary depends on his not understanding it.”
Note the word "difficult," not impossible. This is Sinclair's view and I think it's correct. Getting people to believe (true) things against their material interests to believe is possible but the skill level required is higher than a neutral presentation of the facts to a neutral third party.
Interestingly, the Motivated Cognition framing suggests that there might not be a complete truth of the matter for whether "Mistake Theory" vs "Conflict Theory" vs Negotiation Theory is more correct for a given political disagreement. Instead, your preferred framing has a viewpoint-dependent and normative element to it.
Suppose your objective is just to get a specific policy passed (no meta-level preferences like altruism), and you believe this policy is in your interests and those of many others, and people who oppose you are factually wrong.
Someone who's suited to explanations like Scott (or like me?), might naturally fall into a Mistake Theory framing, and write clear-headed blogposts about why people who disagree with you are wrong. If the Motivated Cognition theory is correct, most people are at least somewhat sincere, and at some level of sufficiently high level of simplicity, people can update to agree with you even if it's not immediately in their interests (smart people in democracies usually don't believe 2+2=5 even in situations where it'd be advantageous for them to do so)
Someone who's good at negotiations and cooperative politics might more naturally adopt a Negotiation Theory framing, and come up with a deal that gets everybody (or enough people) what they want while having their preferred policy passed.
Finally, someone who's good at (or temperamentally suited to) non-cooperative politics and the more Machiavellian side of politics might identify the people who are most likely to oppose their preferred policies, and destroy their political influence enough that the preferred policy gets passed.
Anyway, here are my four models of political disagreement (Mistake, Conflict, Negotiation, Motivated Cognition). I definitely don't think these four models (or linear combinations of them) explain all disagreements, or are the only good frames for thinking of disagreement. Excited to hear alternatives [1]!
[1] In particular I'm wondering if there is a distinct case for ideological/memetic theories that operate along a similar level of abstraction as the existing theories, as opposed to thinking of ideologies as primarily given us different goals (which would make them slot in well with all the existing theories except maybe Mistake Theory).
Good idea, I reposted the article itself here: https://forum.effectivealtruism.org/posts/GyenLpfzRKK3wBPyA/the-simple-case-for-ai-catastrophe-in-four-steps
I've been trying to keep the "meta" and the main posts mostly separate so hopefully the discussions for the metas and the main posts aren't as close together.
The bets I've seen you post seem rather disadvantageous to the other side, and I believed so at the time. Which is fine/good business from your perspective given that you managed to find takers. But it means I'm more pessimistic on finding good deals by both of our lights.
From a information-theoretic perspective, people almost never refer to a single data point as strictly as just one bit, so whether you are counting only one float in a database or a whole row in a structured database, or also a whole conversation, we're sort of negotiating price.
I think the "alien seeing a car" makes the case somewhat clearer. If you already have a deep model of cars (or even a shallow one), seeing another instance of a Ford Focus tells you relatively little, but an alien coming across one will get many bits about it, perhaps more than a human spending an afternoon in Vietnam.