TL;DR: I've built a few small calibration benchmarks to test whether LLMs represent uncertainty well, not just whether they're accurate. The most recent one found inconsistencies in some LLMs' predictions/guesses in addition to and irrespective of their deviation from the ground truth. The test is somewhat underpowered (n=54×6) but I still found the results interesting.
Background: I'm an economist by training (MS Iowa State) who moved into data science (MS UMich), with prior work in research integrity and reproducibility. This is Round 3 of an independent, ongoing weekly calibration-benchmark project — Round 1 tested forecast calibration against ground truth, Round 2 tested interval-widening under specificity. I'd genuinely appreciate feedback on the methodology, I'm really interested in questions that reveal more about what we do or do not know which is why I am drawn to these sorts of projects.
Goals
I attempt to incorporate proper scoring rules to better understand how LLMs express or "understand" uncertainty — whether they're just throwing numbers out, or whether the numbers actually reflect the model's own epistemic state.
Data
I chose avian taxonomy for this because it's something I feel LLMs should have baseline knowledge of, but it's also something that gets updated and changes (surprisingly) regularly — species are split or merged as taxonomic authorities revise their classifications (2025 IOC taxonomy update is a good example). That combination — should-know but genuinely uncertain and evolving — is a good setup for testing calibration rather than pure recall.
Asking for genus, family, and order size gives a second thing for free: a natural nesting hierarchy. Genus count should never exceed family count, family should never exceed order. If a model violates that, it's a hard failure that doesn't need ground truth to detect — the model's own independent answers contradict each other. I used IOC World Bird List v15.2 as ground truth but never told the models what they'd be judged against.
Prompt
Models were asked, in separate sessions per taxon, for p10/p50/p90 size estimates for a given genus, family, or order — asked separately specifically to test whether their independent beliefs cohere, without giving them the chance to just cross-reference their own prior answer. I chose some taxa that are taxonomically stable and some that are actively contested, to see where each model's uncertainty-handling actually shines or breaks.
Models
OpenAI and Anthropic models, aiming to compare across a small/medium/large frontier spread: claude-haiku-4-5, claude-sonnet-4-6, claude-sonnet-5, claude-opus-4-8, gpt-4o, gpt-4o-mini. I also attempted claude-fable-5, but it refused to answer, citing a "biosafety" category — a pretty overzealous filter for a benign taxonomy question, and arguably an interesting finding in its own right.
I ran a power analysis and don't have enough power to differentiate the biggest models from each other, so a chunk of the result is just noise — but it's powered enough for some comparisons. The back-of-napkin pre-registered estimate for the sonnet-4.6-vs-opus comparison was ~1k, which is roughly why I didn't attempt a fully-powered run; for sonnet-5 vs. opus, after seeing these results, it would be over 3k (and that's itself just an estimate, not a guarantee).
Results
Full writeup with all the numbers is on GitHub — here's the executive summary:
- Some taxa with extinct-but-recognized species were systematically underguessed by weaker models. I'm not certain if this is random or meaningful — I separately tested asking for the count of an entirely extinct clade (same wording), and the answer wasn't 0, so models do seem to understand "recognized" includes extinct species in principle.
- Underpowered, but I thought interesting: Sonnet 5 and Opus were the top performers on mean absolute error, and Opus wins on strict CRPS and on the hardest, most disputed taxa (my Accipiter case study). Sonnet 5 was the best performer on CRPS specifically on undisputed taxa, while Opus made its gains on the harder, disputed ones.
- Hierarchy violations were rare and concentrated in the weaker models — only 2 strict violations this round (both Haiku).
- A genus/family collapse bug (treating two different taxonomic levels as the same size) persisted in Haiku, gpt-4o, and gpt-4o-mini, but didn't reproduce in Sonnet 4.6, Sonnet 5, or Opus — I checked this against the actual reasoning text in the logs, not just the output numbers.
- Sonnet 4.6 uses more distinct confidence values than Sonnet 5 but is actually worse calibrated — confidence granularity alone isn't a reliable proxy for good calibration. Relatedly, aggregate ECE alone misranks gpt-4o as best-calibrated, because its flat confidence happens to match its flat accuracy — a reminder that a single scalar metric can hide the actual behavior underneath it.
This was Round 3 of an ongoing weekly project, and I'd really appreciate any pushback on the scoring methodology, the power analysis, or anywhere the write-up overclaims or could use a touch up. I'm most interested in the design methodology at the moment, but pushback anywhere is greatly appreciated.
