Cause prioritizationCause prioritization

This is a linkpost for https://heresjonny.substack.com/p/paul-christianos-barista. 

When I left my previous job in April of last year, I took a long break from thinking about AI safety. This was nice. Thinking about catastrophic risks all the time wears you down. Imagining oneself as Cassandra, baffled that nobody around you is taking this sufficiently seriously, is not good for the soul^[1]. I was angry at the community that had got me interested in AI risk in the first place. I felt powerless to help, and that I had earned more selfishness anyway.

Of course, avoiding AI altogether wasn’t an option. I was worried that coding would soon be automated, and my most marketable skill would disappear. I wanted to broaden my skillset and position myself well in case I couldn’t go back to a regular software engineering job. I started writing posts about how I was going to found a company, after having been heavily influenced by reading The Intelligence Curse^[2]. I got really into thinking about the intersection of AI and psychiatry. I started talking about the importance of defensive acceleration and how effective altruists systematically underrate comparative advantage. This was motivated reasoning; I wanted to work on something other than AI safety and not feel guilty about it^[3]. I figured that all these super smart, status-y types had it under control already.

                                          The EMH is trivially false across all domains

In April of this year, I had various conversations that led to me feeling excited about being part of the EA community again. I went to EAG London and primarily talked about things other than AI safety^[4]. I started doing some contract work for an EA-flavoured global health and development charity. I was managing to feel like I was part of the community without getting sucked into AI safety’s gravity well.

I ended up applying for a job at an EA organisation. I managed to get through to the work trial stage, which involved giving people advice. It was noteworthy that diverse groups of people often seemed to need to hear relatively similar advice. I kept repeating something along the lines of:

Build a model of the world. Try to falsify the components of that model. Seriously think through the implications of this model, and then act accordingly.

And then of course something deeply upsetting happened. I realised that I was not following my own advice.

So I went back to basics. Claude and I went back-and-forth over my litany of questions and objections to the idea that advanced AI systems could pose an existential threat. I wrote a post where I attempted to reason through what I thought was likely to happen over the next two years. It started to seem likely that, at the very least, things are going to get weird, soon.

Somewhere during the process it dawned on me that vanishingly few people do this kind of thing; most were LARPing, just like me. When considering something complex, at some point, it’s all too easy to let rigour lapse and hand-wave something away. Thinking rigorously is hard, but in a particularly subtle way that makes this kind of hand-waving easily done without being consciously noticeable.

I felt a sense of clarity. It might be that this clarity is entirely fake and I’m totally wrong, but still, it persists. I was ready to say it with roon:

experts are fake, smart generalists rule the world, everything is designed by people no smarter than you, courage is in shorter supply than genius

I’ve long felt like I’m trapped in a midwit meme, painfully aware of the importance of AI safety but simply lacking the intelligence or skills to actually do anything about it. When I read Scott Alexander’s Parable of the Talents, I imagined myself as Paul Christiano’s personal barista. But I don’t think any of it is true. I’ve sufficient genius. What I was lacking was the courage to actually do the hard work of thinking seriously about things for long enough to reach a conclusion, even when that conclusion is an unpleasant one.

Having sat down and made a good-faith effort to think it through, I do not think we are on track to develop AGI in a safe way. I think I have enough power and resources that there’s some chance I could make a difference. And so I’m going to try.

1. ^{^}

   Although Michael Burry seems to be making it work for him

2. ^{^}

   I feel weird about this. In some sense, it is true; I am the sole director of a company. The word “found” has various implications that aren’t accurate in my case, but sometimes it is useful to pretend that they are. Sometimes “founding” is quite high-status, and so there’s a temptation to play it up. I think that’s pretty clearly what was happening in this post. The whole thing seems naïve too – although I think that finding success with a startup probably does require some amount of naïveté.

3. ^{^}

   Despite being motivated, I do think there’s some legitimacy to the claim regardless.

^{^}

It was my sixth EAG, and the other five had all been dominated by the topic.

Show all footnotes

EA often gets criticized for taking small probabilities too seriously. But people take actions with small probabilities of big success all the time! Asking a girl out? Small chance she'll become your wife. Trying to affect political chance? Fat chance of that. Launching a startup? Good luck buddy. The difference between EA and the standard way of acting is that standard psychology deals with the problem of small probabilities by being highly overconfident. Of course _my_ startup/political movement/YouTube channel is going to make it big. You cannot maintain high agency while ignoring small probability options. 

The EA approach is simply more honest. 

Help The Unjournal prioritize AI economics and governance research

(https://uj-prioritization-dashboard.netlify.app/ai-governance/)

The Unjournal (https://info.unjournal.org/) is prioritizing research on AI economics, governance, social and economic impacts, and risks from increasingly capable AI. (NB: We're not covering technical/CS/ML/AI-safety research.). We want to commission expert public evaluations and help synthesize, disseminate, and curate this work.

Much of this research seems 1) important/impactful/influential, (2) involves considerable 'firepower' from prominent researchers and practitioners, (3) not 'obviously true', and (4) time-sensitive: relevant for funding, policy, and research-steering now, but not after the "six months to six years" required for the traditional (economics) journal process

Our preliminary shortlist page (https://uj-prioritization-dashboard.netlify.app/ai-governance/) includes paper summaries, AI-generated audio walkthroughs, and an interface for giving feedback. We also invite suggestions fr other research (please check the broader list/interface to avoid duplication).

Especially looking for your input on:

• Usefulness and accuracy:* Is this useful? How could we improve it? What have we got wrong?
• Impact: Which research do you use, or think others use or should use?
• Scrutiny What claims and approaches need careful experts and practitioners critique? What sorts of experts? If you're that expert, please dm and join our pool.
• Participation: Who should be part of this discussion (nominate and nudge)?
• Gaps: What's missing (research, claims, etc.)?

Timing and next steps (our plan)

• Finalize an initial shortlist of at least three research outputs before the end of October, with more depending on interest and funding.
• Commission rapid, public, impact-focused evaluations/reviews by experts and practitioners, aiming for a first public evaluation within about two weeks of commissioning.
• Over the following two to four weeks, invite author and stakeholder responses and follow-ups, and facilitate private and public discussion

If it this works well, we hope to scale up and continue in the following months.

Alongside the evaluations, we want to help people find and use this research, and to help shape its direction. I hope the prioritization itself will be useful to students, researchers, practitioners, and other stakeholders. (NB: the "prioritization" here is mainly about "potential for impact and value for evaluation"; assessing the research credibility is for the evaluation part).

I wanted to share this quickly; planning a fuller post soon. (Feedback on this post itself is helpful too.)

Related: We’re also exploring sponsorship of evaluations/themes/questions (https://uj-prioritization-dashboard.netlify.app/sponsor/). Looking for feedback: Would this be useful, would you sponsor or recommend others to do so? Would sponsorship affect your trust in the evaluations (given the rules and safeguards mentioned in the link)?

At my particular place, I think that improving AI safety community epistemics (writing thorough and careful arguments against people both in the community and outside of it who make silly mistakes) is, by most reasonable estimates, probably less impactful than helping to run MAIA, which is what I currently spend most of my time doing.

I don’t think I’m the only person in this situation, which is quite scary to me, as I think there are a lot of things are epistemically iffy in the AI safety community. For example:

• A lot of the most viral pro-AIS people (especially the ones that are relatively new to the field) seem to not be reasoning especially carefully
• The field’s hiring process is quite insular
• Not enough time is spent engaging with a lot of good faith disagreements (imo)
• Fieldbuilding strategy doesn’t optimize for careful reasoning amongst those it recruits
• People receive lots of social/material benefits for continuing to believe that AI risk is the most important thing and work on it
• Even under some of the shortest timelines and most speculative capabilities that superintelligence might have, I really don’t extinction within the next year should be assigned any notable probability mass, due to the need for data centers to be replenished

I also see a lot of disagreements that I think are poorly reasoned. For example: 

• Poor application of/thinking about EV when conceptualizing risks
• Psychoanalysis that doesn’t explain the safety community’s behaviors well
• Comments significantly overstating the case for something
• Arguments without the slightest amount of research into what the other side actually believes

Tbh though, most of these critiques apply to both sides.

I think this primarily arises because the people working in AI safety who are the most careful reasoners also have the highest opportunity costs, because they tend to be good at other things. This means it’s less valuable for them to write out their thoughts online carefully.

My main point, though, is that it’s a little scary to see so much going wrong with the discourse and not being able to do anything about it (or even be worth trying to do anything about it). It feels like there’s not much worth trying except praying for Forethought to do a good job and reach everyone.

As a new person here, i am curious to learn more. I think that Longtermism strongly implies growth-oriented policies, and that might even be the existential risk-minimizing strategy. I did an entire substack post on my thoughts, and would be interested to hear where you might think i am wrong (I probably am, you may have thought a lot more on it than I have, or just be smarter). Also what is the general longtermist take on AI-regulation, and the larger statement made here - We Must Act Now. It seems to me that carte blanche calls to action are not really helpful, but would be open to changing my mind. :-)

https://open.substack.com/pub/ulrikahm/p/longtermism-and-the-moral-necessity?r=4cnrou&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true

In Bruce Friedrich's new book, he writes, "Sometimes when I talk about cultivated meat someone will bring up the handful of states that have banned it. I'm mostly unconcerned. Cultivated meat companies won't be able to supply all 50 US states anytime soon anyway. Once there are multiple companies selling their products in...the majority of cities all across the country, the states that banned will-- I predict-- quietly repeal their laws" (p. 191).

It's hard to know how literally to interpret this apparently sanguine attitude, as the book is designed to generate enthusiasm for alternative proteins. But, still it seems raise an important question about the cost-effectiveness of repealing existing bans or preventing further ones. Initial thoughts:
 

• My guess is that he'd still view preventing additional bans as important, at least in key regions expected to be first-adopters. Maybe Florida and Texas would be laggards in adoption even if cultivated meat were legal.
• It'd be interesting to do an outside view analysis to see how quickly bans on other novel products have been undone once they've achieved a certain level of popularity elsewhere.
• He's writing as if the industry can definitely succeed in spite of the bans. But, even if the bans spread no further, they already apply to >140M potential consumers across the US and Europe. That, combined with uncertainty about the prospect of additional bans, may chill the sort of public and private investment necessary for industry success.
   

A bird's eye view on why donating to relieve the earthquake's damage to Venezuela is one of the best causes to donate to.

I have seen the photos, tens of complete buildings shattered to pieces, more than 150+ reported dead, and a lot more buried without clue to whether they're dead or alive, and on top of that, a poor, government in crisis country has to handle that. When a natural catastrophe happens to a country like that, your money goes a long way in saving lives. 

Bettering social media platforms could be a top-tier systemic intervention.

My understanding is that EA puts most systemic change / solutions S-rated in terms of impact but F-rated in terms of neglectedness and tractability.

Is this the case for social media? Is it super hard and attempted to make the platforms, either internally or through policy, to not maximize for engagement? What about popularizing (and building if necessary) platforms that reward for what would be "useful" for people (or society at large)?

Because I think it's the most impactful systemic leverage I can think of. Unlike reforms to traditional media, entertainment, education and even politics, changes to them would reach and make at least small changes to most most of the demographics of most countries. And we need to add to this that they are not just replacing those sources of influence to individuals (except education I guess?), but also replacing the last bastion of influence: social circles.