Recent generations of Claude seem better at understanding blog posts and making fairly subtle judgment calls than most smart humans. These days when I’d read an article that presumably sounds reasonable to most people but has what seems to me to be a glaring conceptual mistake, I can put it in Claude, ask it to identify the mistake, and more likely than not Claude would land on the same mistake as the one I identified. I think before Opus 4 this was essentially impossible, Claude 3.xs can sometimes identify small errors but it’s a crapshoot on whether it can identify central mistakes, and certainly not judge it well. It’s possible I’m wrong about the mistakes here and Claude’s just being sycophantic and identifying which things I’d regard as the central mistake, but if that’s true in some ways it’s even more impressive. Interestingly, both Gemini and ChatGPT failed at these tasks. (They can sometimes directionally approach the error I identified, but their formulation is imprecise and broad, and they only have it in a longer list of potential quibbles rather than zero in on the most damning issue).  For clarity purposes, here are 3 articles I recently asked Claude to reassess (Claude got the central error in 2/3 of them). I'm also a little curious what the LW baseline is here; I did not include my comments in my prompts to Claude. https://terrancraft.com/2021/03/21/zvx-the-effects-of-scouting-pillars/ https://www.clearerthinking.org/post/what-can-a-single-data-point-teach-you https://www.lesswrong.com/posts/vZcXAc6txvJDanQ4F/the-median-researcher-problem-1 

Why don’t EA chapters exist at very prestigious high schools (e.g., Stuyvesant, Exeter, etc.)? It seems like a relatively low-cost intervention (especially compared to something like Atlas), and these schools produce unusually strong outcomes. There’s also probably less competition than at universities for building genuinely high-quality intellectual clubs (this could totally be wrong).

Here’s a random org/project idea: hire full-time, thoughtful EA/AIS red teamers whose job is to seriously critique parts of the ecosystem — whether that’s the importance of certain interventions, movement culture, or philosophical assumptions. Think engaging with critics or adjacent thinkers (e.g., David Thorstad, Titotal, Tyler Cowen) and translating strong outside critiques into actionable internal feedback. The key design feature would be incentives: instead of paying for generic criticism, red teamers receive rolling “finder’s fees” for critiques that are judged to be high-quality, good-faith, and decision-relevant (e.g., identifying strategic blind spots, diagnosing vibe shifts that can be corrected, or clarifying philosophical cruxes that affect priorities). Part of why I think this is important is because I generally think have the intuition that the marginal thoughtful contrarian is often more valuable than the marginal agreer, yet most movement funding and prestige flows toward builders rather than structured internal critics. If that’s true, a standing red-team org — or at least a permanent prize mechanism — could be unusually cost-effective. There have been episodic versions of this (e.g., red-teaming contests, some longtermist critiquing stuff), but I’m not sure why this should come in waves rather than exist as ongoing infrastructure (org or just some prize pool that's always open for sufficiently good criticisms).

Is the recent partial lifting of US chip export controls on China (see e.g. here: https://thezvi.substack.com/p/selling-h200s-to-china-is-unwise) good or bad for humanity? I’ve seen many takes from people whose judgment I respect arguing that it is very bad, but their arguments, imho, just don’t make sense. What am I missing? For transparency, I am neither Chinese nor American, nor am I a paid agent of them. I am not at all confident in this take, but imho someone should make it. I see two possible scenarios: A) you are not sure how close humanity is to developing superintelligence in the Yudkowskian sense. This is what I believe, and what many smart opponents of the Trump administration’s move to ease chip controls believe. Or B) you are pretty sure that humanity is not going to develop superintelligence any time soon, let’s say in the next century. I admit that the case against the lifting of chip controls is stronger under B), though I am ultimately inclined to reject it in both scenarios. Why is easing of chip controls, imho, a good idea if the timeline to superintelligence might be short? If superintelligence is around the corner, here is what should be done: an immediate international pause of AI development until we figure out how to proceed. Competitive pressures and resulting prisoner’s dilemmas have been identified as the factor that might push us toward NOT pausing even when it would be widely recognized that the likely outcome of continuing is dire. There are various relevant forms of competition, but plausibly the most important is that between the US and China. In order to reduce competitive dynamics and thus prepare the ground for a cooperative pause, it is important to build trust between the parties and beware of steps that are hostile, especially in domains touching AI. Controls make sense only if you are very confident that superintelligence developed in the US, or perhaps in liberal democracy more generally, is going to turn out well for h

"Most people make the mistake of generalizing from a single data point. Or at least, I do." - SA When can you learn a lot from one data point? People, especially stats- or science- brained people, are often confused about this, and frequently give answers that (imo) are the opposite of useful. Eg they say that usually you can’t know much but if you know a lot about the meta-structure of your distribution (eg you’re interested in the mean of a distribution with low variance), sometimes a single data point can be a significant update. This type of limited conclusion on the face of it looks epistemically humble, but in practice it's the opposite of correct. Single data points aren’t particularly useful when you know a lot, but they’re very useful when you have very little knowledge to begin with. If your uncertainty about a variable in question spans many orders of magnitude, the first observation can often reduce more uncertainty than the next 2-10 observations put together[1]. Put another way, the most useful situations for updating massively from a single data point are when you know very little to begin with. For example, if an alien sees a human car for the first time, the alien can make massive updates on many different things regarding Earthling society, technology, biology and culture. Similarly, an anthropologist landing on an island of a previously uncontacted tribe can rapidly learn so much about a new culture from a single hour of peaceful interaction [2]. Some other examples: * Your first day at a new job. * First time visiting a country/region you previously knew nothing about. One afternoon in Vietnam tells you roughly how much things cost, how traffic works, what the food is like, languages people speak, how people interact with strangers. * Trying a new fruit for the first time. One bite of durian tells you an enormous amount about whether you'll like durian. * Your first interaction with someone's kid tells you roughly how old they are, how v