Quick takes

Not that we can do much about it, but I find the idea of Trump being president in a time that we're getting closer and closer to AGI pretty terrifying. A second Trump term is going to have a lot more craziness and far fewer checks on his power, and I expect it would have significant effects on the global trajectory of AI.

As someone predisposed to like modeling, the key takeaway I got from Justin Sandefur's Asterisk essay PEPFAR and the Costs of Cost-Benefit Analysis was this corrective reminder – emphasis mine, focusing on what changed my mind: More detail: Tangentially, I suspect this sort of attitude (Iraq invasion notwithstanding) would naturally arise out of a definite optimism mindset (that essay by Dan Wang is incidentally a great read; his follow-up is more comprehensive and clearly argued, but I prefer the original for inspiration). It seems to me that Justin has this mindset as well, cf. his analogy to climate change in comparing economists' carbon taxes and cap-and-trade schemes vs progressive activists pushing for green tech investment to bend the cost curve. He concludes:  Aside from his climate change example above, I'd be curious to know what other domains economists are making analytical mistakes in w.r.t. cost-benefit modeling, since I'm probably predisposed to making the same kinds of mistakes. 

This December is the last month unlimited Manifold Markets currency redemptions for donations are assured: https://manifoldmarkets.notion.site/The-New-Deal-for-Manifold-s-Charity-Program-1527421b89224370a30dc1c7820c23ec Highly recommend redeeming donations this month since there are orders of magnitude more currency outstanding than can be donated in future months

Metaculus launches round 2 of the Chinese AI Chips Tournament Help bring clarity to key questions in AI governance and support research by the Institute for AI Policy and Strategy (IAPS). Start forecasting on new questions tackling broader themes of Chinese AI capability like:  Will we see a frontier Chinese AI model before 2027? Will a Chinese firm order a large number of domestic AI chips? Will a Chinese firm order a large number of US or US-allied AI chips?

Metaculus Concludes Nuclear Risk Tournament Supporting Rethink Priorities Metaculus has concluded the Nuclear Risk Tournament, supporting Rethink Priorities' efforts by helping to inform funding, policy, research, and career decisions aimed at reducing existential risk.  Thank you to Rethink Priorities for sponsoring the tournament, thank you to the forecasters who contributed their talents, and congratulations to the tournament winners.  Learn more

[Question] How should we think about the decision relevance of models estimating p(doom)? (Epistemic status: confused & dissatisfied by what I've seen published, but haven't spent more than a few hours looking. Question motivated by Open Philanthropy's AI Worldviews Contest; this comment thread asking how OP updated reminded me of my dissatisfaction. I've asked this before on LW but got no response; curious to retry, hence repost)  To illustrate what I mean, switching from p(doom) to timelines:  * The recent post AGI Timelines in Governance: Different Strategies for Different Timeframes was useful to me in pushing back against Miles Brundage's argument that "timeline discourse might be overrated", by showing how choice of actions (in particular in the AI governance context) really does depend on whether we think that AGI will be developed in ~5-10 years or after that.  * A separate takeaway of mine is that decision-relevant estimation "granularity" need not be that fine-grained, and in fact is not relevant beyond simply "before or after ~2030" (again in the AI governance context).  * Finally, that post was useful to me in simply concretely specifying which actions are influenced by timelines estimates.   Question: Is there something like this for p(doom) estimates? More specifically, following the above points as pushback against the strawman(?) that "p(doom) discourse, including rigorous modeling of it, is overrated": 1. What concrete high-level actions do most alignment researchers agree are influenced by p(doom) estimates, and would benefit from more rigorous modeling (vs just best guesses, even by top researchers e.g. Paul Christiano's views)? 2. What's the right level of granularity for estimating p(doom) from a decision-relevant perspective? Is it just a single bit ("below or above some threshold X%") like estimating timelines for AI governance strategy, or OOM (e.g. 0.1% vs 1% vs 10% vs >50%), or something else? * I suppose the easy answer is "t

TL;DR: Someone should probably write a grant to produce a spreadsheet/dataset of past instances where people claimed a new technology would lead to societal catastrophe, with variables such as “multiple people working on the tech believed it was dangerous.” Slightly longer TL;DR: Some AI risk skeptics are mocking people who believe AI could threaten humanity’s existence, saying that many people in the past predicted doom from some new tech. There is seemingly no dataset which lists and evaluates such past instances of “tech doomers.” It seems somewhat ridiculous* to me that nobody has grant-funded a researcher to put together a dataset with variables such as “multiple people working on the technology thought it could be very bad for society.” *Low confidence: could totally change my mind  ——— I have asked multiple people in the AI safety space if they were aware of any kind of "dataset for past predictions of doom (from new technology)", but have not encountered such a project. There have been some articles and arguments floating around recently such as "Tech Panics, Generative AI, and the Need for Regulatory Caution", in which skeptics say we shouldn't worry about AI x-risk because there are many past cases where people in society made overblown claims that some new technology (e.g., bicycles, electricity) would be disastrous for society. While I think it's right to consider the "outside view" on these kinds of things, I think that most of these claims 1) ignore examples of where there were legitimate reasons to fear the technology (e.g., nuclear weapons, maybe synthetic biology?), and 2) imply the current worries about AI are about as baseless as claims like "electricity will destroy society," whereas I would argue that the claim "AI x-risk is >1%" stands up quite well against most current scrutiny. (These claims also ignore the anthropic argument/survivor bias—that if they ever were right about doom we wouldn't be around to observe it—but this is less impor

For a long time I found this surprisingly nonintuitive, so I made a spreadsheet that did it, which then expanded into some other things. * Spreadsheet here, which has four tabs based on different views on how best to pick the fair place to bet where you and someone else disagree. (The fourth tab I didn't make at all, it was added by someone (Luke Sabor) who was passionate about the standard deviation method!)  * People have different beliefs / intuitions about what's fair! * An alternative to the mean probability would be to use the product of the odds ratios. Then if one person thinks .9 and the other .99, the "fair bet" will have implied probability more than .945. *  The problem with using Geometric mean can be highlighted if player 1 estimates 0.99 and player 2 estimates 0.01. This would actually lead player 2 to contribute ~90% of the bet for an EV of 0.09, while player 1 contributes ~10% for an EV of 0.89. I don't like that bet. In this case, mean prob and Z-score mean both agree at 50% contribution and equal EVs. * "The tradeoff here is that using Mean Prob gives equal expected values (see underlined bit), but I don't feel it accurately reflects "put your money where your mouth is". If you're 100 times more confident than the other player, you should be willing to put up 100 times more money. In the Mean prob case, me being 100 times more confident only leads me to put up 20 times the amount of money, even though expected values are more equal." * Then I ended up making an explainer video because I was excited about it   Other spreadsheets I've seen in the space: * Brier score betting (a fifth way to figure out the correct bet ratio!) * Posterior Forecast Calculator * Inferring Probabilities from PredictIt Prices These three all by William Kiely. Does anyone else know of any? Or want to argue for one method over another?