Epistemic status: writing fast and loose, but based on thoughts I've mulled over for a while given personal experience/struggles and discussions with other people. Thus, easy to misinterpret what I'm saying. Take with salt.

On the topic of educational choice, I can't emphasize enough the importance of having legible hard skills such as language or, perhaps more importantly, quantitative skills. Perhaps the worst mistake I made in college was choosing to double major in both international studies and public policy, rather than adding a second major in econ or computer science. To my freshman self, this seemed to make sense: "I really want to do international security and like public policy, so this should demonstrate my enthusiasm and improve my policy-analysis skills."

I'm somewhat skeptical that the second belief panned out (relative to majoring in econ), but the first clearly seems to have been misguided. Understanding what I do now about the application process, including how shallow/ineffective the process is at discerning capabilities and interest, how people will just exaggerate/BS their way into positions (through claims in their interviews and cover letters), that some positions will not consider people without STEM backgrounds (even when the skills can sometimes be learned prior to the job), AND how much of the process in some places relies on connections or prestige, it's really clear that having legible hard skills is crucial. In contrast, you probably get rapidly-diminishing marginal returns with soft science degrees.

Ultimately, I've found that the line between empirical and theoretical analysis is often very blurry, and if someone does develop a decent brightline to distinguish the two, it turns out that there are often still plenty of valuable theoretical methods, and some of the empirical methods can be very misleading. 

For example, high-fidelity simulations are arguably theoretical under most definitions, but they can be far more accurate than empirical tests.

Overall, I tend to be quite supportive of using whatever empirical evidence we can, especially experimental methods when they are possible, but there are many situations where we cannot do this. (I've written more on this here: https://georgetownsecuritystudiesreview.org/2022/11/30/complexity-demands-adaptation-two-proposals-for-facilitating-better-debate-in-international-relations-and-conflict-research/ )

I see. (For others' reference, those two points are pasted below)

1. All knowledge is derived from impressions of the external world. Our ability to reason is limited, particularly about ideas of cause and effect with limited empirical experience.
2. History shows that societies develop in an emergent process, evolving like an organism into an unknown and unknowable future. History was shaped less by far-seeing individuals informed by reason than by contexts which were far too complex to realize at the time.

Overall, I don't really know what to make of these. They are fairly vague statements, making them very liable to motte-and-bailey interpretations; they border on deepities, in my reading. 

"All knowledge is derived from impressions of the external world" might be true in a trivially obvious sense that you often need at least some iota of external information to develop accurate beliefs or effective actions (although even this might be somewhat untrue with regard to biological instincts). However, it makes no clear claim about how much and what kind of "impressions from the external world" are necessary for "knowledge."^[1] Insofar as the claim is that forecasts about AI x-risks are not "derived from impressions of the external world," I think this is completely untrue. In such an interpretation, I question whether the principle even lives up to its own claims: what empirical evidence was this claim derived from?

The second claim suffers from similar problems in my view: I obviously wouldn't claim that there have always been seers who could just divine the long-run future. However, insofar as it is saying that the future is so "unknowable" that people cannot reason about what actions in front of them are good, I also reject this: it seems obviously untrue with regards to, e.g., fighting Nazi Germany in WW2. Moreover, I would say that even if this has been true, that does not mean it will always be true, especially given the potential for value lock-in from superintelligent AI. 

Overall, I agree that it's important to be humble about our forecasts and that we should be actively searching for more information and methods to improve our accuracy, questioning our biases, etc. But I also don't trust vague statements that could be interpreted as saying it's largely hopeless to make decision-informing predictions about what to do in the short term to increase the chance of making the long-run future go well.

1. ^{^}

   A term I generally dislike for its ambiguity and philosophical denotations (which IMO are often dubious at best).

I haven't looked very hard but the short answer is no, I'm not aware of any posts/articles that specifically address the idea of "methodological overhang" (a phrase I hastily made up and in hindsight realize may not be totally logical) as it relates to AI capabilities.

That being said, I have written about the possibility that our current methods of argumentation and communication could be really suboptimal, here: https://georgetownsecuritystudiesreview.org/2022/11/30/complexity-demands-adaptation-two-proposals-for-facilitating-better-debate-in-international-relations-and-conflict-research/

Is your claim just that people should generally "increase [their] error bars and widen [their] probability distribution"? (I was frustrated by the difficulty of figuring out what this post is actually claiming; it seems like it would benefit from a "I make the following X major claims..." TLDR.)

I probably disagree with your points about empiricism vs. rationalism (on priors that I dislike the way most people approach the two concepts), but I think I agree that most people should substantially widen their "error bars" and be receptive to new information. And it's for precisely that reason which I feel decently confident in saying "most people whose risk estimates are very low (<0.5%) are significantly overconfident." You logically cannot have extremely low probability estimates while also believing "there's a >10% chance that in the future I will justifiably think there is a >5% chance of doom, but right now the evidence tells me the risk is <0.5%."

I think there is plenty of room for debate about what the curve of AI progress/capabilities will look like, and I mostly skimmed the article in about ~5 minutes, but I don't think your post's content justified the title ("exponential AI takeoff is a myth"). "Exponential AI takeoff is currently unsupported" or "the common narrative(s) for exponential AI takeoff is based on flawed premises" are plausible conclusions from this post (even if I don't necessarily agree with them), but I think the original title would require far more compelling arguments to be justified.

(I won't get too deep into this, but I think it's plausible that there is significant "methodological overhang": humans might just struggle to make progress in some fields of research—especially softer sciences and theory-heavy sciences—because principal-agent problems in research plague the accumulation of reliable knowledge through non-experimental methods.) 

TL;DR: Someone should probably write a grant to produce a spreadsheet/dataset of past instances where people claimed a new technology would lead to societal catastrophe, with variables such as “multiple people working on the tech believed it was dangerous.”

Slightly longer TL;DR: Some AI risk skeptics are mocking people who believe AI could threaten humanity’s existence, saying that many people in the past predicted doom from some new tech. There is seemingly no dataset which lists and evaluates such past instances of “tech doomers.” It seems somewhat ridiculous* to me that nobody has grant-funded a researcher to put together a dataset with variables such as “multiple people working on the technology thought it could be very bad for society.”

*Low confidence: could totally change my mind 

———

I have asked multiple people in the AI safety space if they were aware of any kind of "dataset for past predictions of doom (from new technology)"? There have been some articles and arguments floating around recently such as "Tech Panics, Generative AI, and the Need for Regulatory Caution", in which skeptics say we shouldn't worry about AI x-risk because there are many past cases where people in society made overblown claims that some new technology (e.g., bicycles, electricity) would be disastrous for society.

While I think it's right to consider the "outside view" on these kinds of things, I think that most of these claims 1) ignore examples of where there were legitimate reasons to fear the technology (e.g., nuclear weapons, maybe synthetic biology?), and 2) imply the current worries about AI are about as baseless as claims like "electricity will destroy society," whereas I would argue that the claim "AI x-risk is >1%" stands up quite well against most current scrutiny.

(These claims also ignore the anthropic argument/survivor bias—that if they ever were right about doom we wouldn't be around to observe it—but this is less important.)

I especially would like to see a dataset that tracks things like "were the people warning of the risks also the people who were building the technology?" More generally, some measurement of "analytical rigor" also seems really important, e.g., "could the claims have stood up to an ounce of contemporary scrutiny (i.e., without the benefit of hindsight)?"

Absolutely seems worth spending up to $20K to hire researchers to produce such a spreadsheet within the next two-ish months… this could be a critical time period, where people are more receptive to new arguments/responses…?

TBH, I think that the time spent scoring rationales is probably quite manageable: I don’t think it should take longer than 30 person-minutes to decently judge each rationale (e.g., have three judges each spend 10 minutes evaluating each), maybe less? It might be difficult to have results within 1-2 hours if you don’t have that many judges, but probably it should be available by the end of the day.

To be clear, I was thinking that only a small number (no more than three, maybe just two) of the total questions should be “rationale questions.”

But definitely the information value of “do rationale scores correlate with performance” would be interesting! I’m not sure if the literature has ever done this (I don’t think I’ve encountered anything like that, but I haven’t actively searched for it)

I would also strongly recommend having a version of the fellowship that aligns with US university schedules, unlike the current Summer fellowship!

Given the (accusations of) conflicts of interest in OpenAI’s calls for regulation of AI, I would be quite averse to relying on OpenAI for funding for AI governance