Existential risk
Existential risk
Discussions of risks which threaten the destruction of the long-term potential of life

Shortforms

41
15d
2
In Twitter and elsewhere, I've seen a bunch of people argue that AI company execs and academics are only talking about AI existential risk because they want to manufacture concern to increase investments and/or as a distraction away from near-term risks and/or regulatory capture. This is obviously false.  However, there is a nearby argument that is likely true: which is that incentives drive how people talk about AI risk, as well as which specific regulations or interventions they ask for. This is likely to happen both explicitly and unconsciously. It's important (as always) to have extremely solid epistemics, and understand that even apparent allies may have (large) degrees of self-interest and motivated reasoning.  Safety-washing [https://forum.effectivealtruism.org/posts/f2qojPr8NaMPo2KJC/beware-safety-washing] is a significant concern; similar things have happened a bunch in other fields, it likely has already happened a bunch in AI, and will likely happen again in the months and years to come, especially if/as policymakers and/or the general public become increasingly uneasy about AI.
22
18d
TL;DR: Someone should probably write a grant to produce a spreadsheet/dataset of past instances where people claimed a new technology would lead to societal catastrophe, with variables such as “multiple people working on the tech believed it was dangerous.” Slightly longer TL;DR: Some AI risk skeptics are mocking people who believe AI could threaten humanity’s existence, saying that many people in the past predicted doom from some new tech. There is seemingly no dataset which lists and evaluates such past instances of “tech doomers.” It seems somewhat ridiculous* to me that nobody has grant-funded a researcher to put together a dataset with variables such as “multiple people working on the technology thought it could be very bad for society.” *Low confidence: could totally change my mind  ——— I have asked multiple people in the AI safety space if they were aware of any kind of "dataset for past predictions of doom (from new technology)"? There have been some articles and arguments floating around recently such as "Tech Panics, Generative AI, and the Need for Regulatory Caution [https://datainnovation.org/2023/05/tech-panics-generative-ai-and-regulatory-caution/]", in which skeptics say we shouldn't worry about AI x-risk because there are many past cases where people in society made overblown claims that some new technology (e.g., bicycles, electricity) would be disastrous for society. While I think it's right to consider the "outside view" on these kinds of things, I think that most of these claims 1) ignore examples of where there were legitimate reasons to fear the technology (e.g., nuclear weapons, maybe synthetic biology?), and 2) imply the current worries about AI are about as baseless as claims like "electricity will destroy society," whereas I would argue that the claim "AI x-risk is >1%" stands up quite well against most current scrutiny. (These claims also ignore the anthropic argument/survivor bias—that if they ever were right about doom we wouldn't
6
11d
Scattered and rambly note I jotted down in a slack in February 2023, and didn't really follow up on -------------------------------------------------------------------------------- thinking of jotting down some notes about "what AI pessimism funding ought to be", that takes into account forecasting and values disagreements.The premises:   * threatmodels drive research. This is true on lesswrong when everyone knows it and agonizes over "am I splitting my time between hard math/cs and forecasting or thinking about theories of change correctly?" and it's true in academia when people halfass a "practical applications" paragraph in their paper. * people who don't really buy into the threatmodel they're ostensibly working on do research poorly * social pressures like funding and status make it hard to be honest about what threatmodels motivate you. * I don't overrate democracy or fairness as terminal values, I'm bullish on a lot of deference and technocracy (whatever that means), but I may be feeling some virtue-ethicsy attraction toward "people feeling basically represented by governance bodies that represent them", that I think is tactically useful for researchers because the above point about research outputs being more useful when the motivation is clearheaded and honest. * fact-value orthogonality, additionally the binary is good and we don't need a secret third thing if we confront uncertainty well enough The problems I want to solve:   * thinking about inclusion and exclusion (into "colleagueness" or stuff that funder's care about like "who do I fund") is fogged by tribal conflict where people pathologize eachother (salient in "AI ethics vs. AI alignment". twitter is the mindkiller but occasionally I'll visit, and I always feel like it makes me think less clearly) * no actual set of standards for disagreement to take place in, instead we have wishy washy stuff like "the purple hats undervalue standpoint
17
1mo
I'm thinking about the matching problem of "people with AI safety questions" and "people with AI safety answers". Snoop Dogg hears Geoff Hinton on CNN (or wherever), asks "what the fuck?" [https://twitter.com/pkedrosky/status/1653955254181068801], and then tries to find someone who can tell him what the fuck. I think normally people trust their local expertise landscape--if they think the CDC is the authority on masks they adopt the CDC's position, if they think their mom group on Facebook is the authority on masks they adopt the mom group's position--but AI risk is weird because it's mostly unclaimed territory in their local expertise landscape. (Snoop also asks "is we in a movie right now?" because movies are basically the only part of the local expertise landscape that has had any opinion on AI so far, for lots of people.) So maybe there's an opportunity here to claim that territory (after all, we've thought about it a lot!). I think we have some 'top experts' who are available for, like, mass-media things (podcasts, blog posts, etc.) and 1-1 conversations with people they're excited to talk to, but are otherwise busy / not interested in fielding ten thousand interview requests. Then I think we have tens (hundreds?) of people who are expert enough to field ten thousand interview requests, given that the standard is "better opinions than whoever they would talk to by default" instead of "speaking to the whole world" or w/e. But just like connecting people who want to pay to learn calculus and people who know calculus and will teach it for money, there's significant gains from trade from having some sort of clearinghouse / place where people can easily meet. Does this already exist? Is anyone trying to make it? (Do you want to make it and need support of some sort?)
1
3d
The following is a conversation between myself in 2022, and a newer version of myself earlier this year. On the Nature of Intelligence and its "True Name": 2022 Me:  This has become less obvious to me as I’ve tried to gain a better understanding of what general intelligence is. Until recently, I always made the assumption that intelligence and agency were the same thing. But General Intelligence, or G, might not be agentic. Agents that behave like RLs may only be narrow forms of intelligence, without generalizability. G might be something closer to a simulator. From my very naive perception of neuroscience, it could be that we (our intelligence) is not agentic, but just simulates agents. In this situation, the prefrontal cortex not only runs simulations to predict its next sensory input, but might also run simulations to predict inputs from other parts of the brain. In this scenario, “desire” or “goals”, might be simulations to better predict narrowly-intelligent agentic optimizers. Though the simulator might be myopic, I think this prediction model allows for non-myopic behavior, in a similar way GPT has non-myopic behavior, despite only trying to predict the next token (it has an understanding of where a future word “should” be within the context of a sentence, paragraph, or story). I think this model of G allows for the appearance of intelligent goal-seeking behavior, long-term planning, and self-awareness. I have yet to find another model for G that allows for all three. The True Name of G might be Algorithm Optimized To Reduce Predictive Loss. 2023 Me: interesting, me’22, but let me ask you something: you seem to think this majestic ‘G’ is something humans have, but other species do not, and then name the True Name of ‘G’ to be Algorithm Optimized To Reduce Predictive Loss. Do you *really* think other animals don’t do this? How long is a cat going to survive if it can’t predict where it’s going to land? Or where the mouse’s path trajectory is heading? Did yo
7
19d
8
Hey - I’d be really keen to hear peoples' thoughts on the following career/education decision I'm considering (esp. people who think about AI a lot): * I’m about to start my undergrad studying PPE at Oxford. * I’m wondering whether re-applying this year to study CS & philosophy at Oxford (while doing my PPE degree) is a good idea. * This doesn’t mean I have to quit PPE or anything.  * I’d also have to start CS & philosophy from scratch the following year. * My current thinking is that I shouldn’t do this - I think it’s unlikely that I’ll be sufficiently good to, say, get into a top 10 ML PhD or anything, so the technical knowledge that I’d need for the AI-related paths I’m considering (policy, research, journalism, maybe software engineering) is either pretty limited (the first three options) or much easier to self-teach and less reliant on credentials (software engineering). * I should also add that I’m currently okay at programming anyway, and plan to develop this alongside my degree regardless of what I do - it seems like a broadly useful skill that’ll also give me more optionality. * I do have a suspicion that I’m being self-limiting re the PhD thing - if everyone else is starting from a (relatively) blank slate, maybe I’d be on equal footing?  * That said, I also have my suspicions that the PhD route is actually my highest-impact option: I’m stuck between 1) deferring to 80K here, and 2) my other feeling that enacting policy/doing policy research might be higher-impact/more tractable. * They’re also obviously super competitive, and seem to only be getting more so. * One major uncertainty I have is whether, for things like policy, a PPE degree (or anything politics-y/economics-y) really matters. I’m a UK citizen, and given the record of UK politicians who did PPE at Oxford, it seems like it might? What mistakes am I making here/am I being too self-limiting? I s
3
10d
Note: this sounds like it was written by chatGPT because it basically [https://audiopen.ai] was (from a recorded ramble)🤷‍   I believe the Forum could benefit from a Shorterform page, as the current Shortform forum, intended to be a more casual and relaxed alternative to main posts, still seems to maintain high standards. This is likely due to the impressive competence of contributors who often submit detailed and well-thought-out content. While some entries are just a few well-written sentences, others resemble blog posts in length and depth. As such, I find myself hesitant to adhere to the default filler text in the submission editor when visiting this page. However, if it were more informal and less intimidating in nature, I'd be inclined to post about various topics that might otherwise seem out of place. To clarify, I'm not suggesting we resort to jokes or low-quality "shitposts," but rather encourage genuine sharing of thoughts without excessive analysis. Perhaps adopting an amusing name like "EA Shorterform" would help create a more laid-back atmosphere for users seeking lighter discussions. By doing so, we may initiate a preference falsification cascade where everyone feels comfortable enough admitting their desire for occasional brevity within conversations. Who knows? Maybe I'll start with posting just one sentence soon!
12
1mo
6
Why aren't we engaging in direct action (including civil disobedience) to pause AI development? Here's the problem: Yudkowksy [https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/]: "Many researchers steeped in these issues, including myself, expect that the most likely result of building a superhumanly smart AI, under anything remotely like the current circumstances, is that literally everyone on Earth will die." Here's one solution: FLI Open Letter [https://futureoflife.org/open-letter/pause-giant-ai-experiments/]: "all AI labs...immediately pause for at least 6 months the training of AI systems more powerful than GPT-4. This pause should be public and verifiable, and include all key actors. If such a pause cannot be enacted quickly, governments should step in and institute a moratorium." Here's what direct action in the pursuit of that solution could look like (most examples are from the UK climate movement): Picketing AI offices [https://twitter.com/Radlib4/status/1653135998501662722?s=20] (this already seems to be happening!) Mass non-disruptive protest [https://www.theguardian.com/world/2023/apr/21/big-one-extinction-rebellion-cliimate-protest-london-xr] Strikes/walk-outs [https://www.theguardian.com/science/2021/sep/24/people-in-99-countries-take-part-in-global-climate-strike] (by AI developers/researchers/academics) Slow marches [https://www.itv.com/news/border/2023-04-29/just-stop-oil-protestors-stage-slow-march-through-town-centre] Roadblocks [https://www.bbc.co.uk/news/uk-england-london-59061509] Occupation [https://www.theguardian.com/uk/2012/jan/16/belfast-occupy-bank-of-ireland] of AI offices Performative vandalism [https://www.bbc.co.uk/news/uk-england-gloucestershire-64193016] of AI offices Performative vandalism of art [https://www.theguardian.com/environment/2022/oct/14/just-stop-oil-activists-throw-soup-at-van-goghs-sunflowers] Sabotage of AI computing infrastructure (on the model of ecotage [https://www.theguardian.
Load more (8/24)