342 karmaJoined


yeah yeah, probably wrong

It's a fib


Thank you for making the effort to write this post. 

Reading Situational Awareness, I updated pretty hardcore into national security as the probable most successful future path, and now find myself a little chastened by your piece, haha [and just went around looking at other responses too, but yours was first and I think it's the most lit/evidence-based]. I think I bought into the "Other" argument for China and authoritarianism, and the ideal scenario of being ahead in a short timeline world so that you don't have to even concern yourself with difficult coordination, or even war, if it happens fast enough. 

I appreciated learning about macrosecuritization and Sears' thesis, if I'm a good scholar I should also look into Sears' historical case studies of national securitization being inferior to macrosecuritization. 

Other notes for me from your article included: Leopold's pretty bad handwaviness around pausing as simply, "not the way", his unwillingness to engage with alternative paths, the danger (and his benefit) of his narrative dominating, and national security actually being more at risk in the scenario where someone is threatening to escape mutually assured destruction. I appreciated the note that safety researchers were pushed out of/disincentivized in the Manhattan Project early and later disempowered further, and that a national security program would probably perpetuate itself even with a lead.


FWIW I think Leopold also comes to the table with a different background and set of assumptions, and I'm confused about this but charitably: I think he does genuinely believe China is the bigger threat versus the intelligence explosion, I don't think he intentionally frames the Other as China to diminish macrosecuritization in the face of AI risk. See next note for more, but yes, again, I agree his piece doesn't have good epistemics when it comes to exploring alternatives, like a pause, and he seems to be doing his darnedest narratively to say the path he describes is The Way (even capitalizing words like this), but...

One additional aspect of Leopold's beliefs that I don't believe is present in your current version of this piece, is that Leopold makes a pretty explicit claim that alignment is solvable and furthermore believes that it could be solved in a matter of months, from p. 101 of Situational Awareness:

Moreover, even if the US squeaks out ahead in the end, the difference between a 1-2 year and 1-2 month lead will really matter for navigating the perils of superintelligence. A 1-2 year lead means at least a reasonable margin to get safety right, and to navigate the extremely volatile period around the intelligence explosion and post-superintelligence.77 [NOTE] 77 E.g., space to take an extra 6 months during the intelligence explosion for alignment research to make sure superintelligence doesn’t go awry, time to stabilize the situation after the invention of some novel WMDs by directing these systems to focus on defensive applications, or simply time for human decision-makers to make the right decisions given an extraordinarily rapid pace of technological change with the advent of superintelligence.

I think this is genuinely a crux he has with the 'doomers', and to a lesser extent the AI safety community in general. He seems highly confident that AI risk is solvable (and will benefit from gov coordination), contingent on there being enough of a lead (which requires us to go faster to produce that lead) and good security (again, increase the lead).

Finally, I'm sympathetic to Leopold writing about the government as better than corporations to be in charge here (and I think the current rate of AI scaling makes this at some point likely (hit proto-natsec level capability before x-risk capability, maybe this plays out on the model gen release schedule)) and his emphasis on security itself seems pretty robustly good (I can thank him for introducing me to the idea of North Korea walking away with AGI weights). Also just the writing is pretty excellent.

Yeah, really interesting thanks for sharing. The incentive structure here seems to be a pretty nice clean loop where a better world model actually does predict more accurately something that matters (better financial news, benefits readers more directly - vs maybe the incentive with other news sources is more meta/abstract - agreeing with your community and being up to date)

And one last thought I have is that the incentives news has to be negative seem to be quite bad. If there were a tractable intervention to mitigate these incentives maybe that could do a lot of good.

And from the ronghosh article: "... All we have to do is fix the loneliness crisis, the fertility crisis, the housing crisis, the obesity crisis, the opioid crisis, the meaning crisis, the meta crisis, the flawed incentives of the political system, the flawed incentives of social media, the flawed incentives of academia, the externalities leading to climate change, the soft wars with China and Iran, the hot war with Russia, income inequality, status inequality, racism, sexism, and every other form of bigotry."

Of course, as someone who's steeped in all the AI stuff, I can't help but just think that A) AI is the most important thing to consider here (ha!), since B) it might allow us ('alignment' allowing) to scale the sort of sense-making and problem-solving cognition to help solve all the problems that we're seemingly increasingly making for ourselves. And yeah this is reductionist and probably naive.

I think the question, "is the world getting better?" is important for effective altruists (soft pitch why is that it's just a crucial consideration for decision-making).

IDK, quick take because I'm just thinking about the following links, and people's perceptions around this question.

170 years of American news coverage: 
https://x.com/DKThomp/status/1803766107532153119 (linked in Marginal Revolution)


"We really are living in an era of negativity-poisoned discourse that is (*empirically*) historically unique."
(and this Atlantic article by the same author as the tweet discussing how America may be producing and exporting a lot of anxiety)

And I thought this piece lays out really quite well the points for both: things are better, things are worse, and introduced me to the neat term, "the vibecession":
(linked in r/slatestarcodex)

In particular, I thought this quote was funny too, and got me: 
"Anecdotally, this is also where a subset of rationalists appear to be inconsistent in their worldview. One moment they claim the majority of people are data illiterate, and are therefore unrealistically pessimistic, and in the next moment they will set p(doom) to 10%."

And [I had more to say here, but I think I'll just leave it to, another excerpt]:
"Like, yes; it’s fairly uncontroversial to say that the world and the economy is better than ever. Even the poorest among us have super computers in our pockets now capable of giving us a never-ending stream of high-quality videos, or the power to summon a car ride, some food, or an Amazon delivery at any given moment.

And yet, all of this growth and change and innovation and wealth has come at the cost of some underlying stability. For a lot of people, they feel like they’re no longer living on land; instead they’ve set sail on a vessel — and the never-ending swaying, however gentle it might feel, is leaving them seasick."

(Finally, I coincidentally also read recently a listicle of ordinary life improvements from Gwern)


Hi JWS, unsure if you’d see this since it’s on LW and I thought you’d be interested (I’m not sure what to think of Chollet’s work tbh and haven’t been able to spend time on it, so I’m not making much of a claim in sharing this!)


FWIW Habryka, I appreciate all that I know you’ve done and expect there’s a lot more I don’t know about that I should be appreciative of too.

I would also appreciate if you’d write up these concerns? I guess I want to know if I should feel similarly even as I rather trust your judgment. Sorry to ask, and thanks again

Editing to note I‘ve now seen some of comments elsewhere

Yeah, thank you, I guess I was trying to say that the evidence only seems to be stronger over time that the Bay Area’s: ‘AI is the only game in town’, is accurate.

Insofar as, timelines for various AI capabilities have outperformed both superforecasters’ and AI insiders’ predictions; transformative AI timelines (at Open Phil, prediction markets, AI experts I think) have decreased significantly over the past few years; the performance of LLMs have increased at an extraordinary rate across benchmarks; and we expect the next decade to extrapolate this scaling to some extent (w/ essentially hundreds of billions if not tens of trillions to be invested).

Although, yeah, I think to some extent we can’t know if this continues to scale as prettily as we’d expect and it’s especially hard to predict categorically new futures like exponential growth (10%, 50%, etc, growth/year). Given the forecasting efforts and trends thus far it feels like there’s a decent chance of these wild futures, and people are kinda updating all the way? Maybe not Open Phil entirely (to the point that EA isn’t just AIS), since they are hedging their altruistic bets, in the face of some possibility this decade could be ‘the precipice’ or one of the most important ever.

Misuse and AI risk seem like the negative valence of AI’s transformational potential. I personally buy the arguments around transformational technologies needing more reasoned steering and safety, and I also buy that EA has probably been a positive influence, and that alignment research has been at least somewhat tractable. Finally I think that there’s more that could be done to safely navigate this transition.

Also, re David (Thorstad?) yeah I haven’t engaged with his stuff as I probably should, and I really don’t know how to reason for or against arguments around the singularity, exponential growth, and the potential of AI without deferring to people more knowledgeable/smarter than me. I do feel like I have seen the start and middle of trends they predicted, and predict will extrapolate-with my own personal use and some early reports on productivity increases.

I do look forward to your sequence and hope you do really well on it!

Load more