As I have been working on SIRA (Societal Impacts Research on AI) for the past three months or so, a common sentiment has been expressed that doesn't sit well with me: that there is limited evidence of current demand for the research and talent SIRA would aim to produce. While I can see how this may be true, I'm reminded of the line attributed (probably apocryphally) to Henry Ford, "If I had asked people what they wanted, they would have said faster horses," and Steve Jobs's version, that people don't know what they want until you show it to them.

I'm using this quick take as a public note to work on this tension and look for evidence against that sentiment. My plan: (1) Make a list of "societal impact" research that I can point to and say, "Hey, look, so-and-so has done this and has addressed xyz." (2) Trace the "impact" of these works to show funders/partners that this type of work can and does make an impact. (3) Write a full blog post about this tension, pointing to this evidence.

This is a linkpost for https://heresjonny.substack.com/p/paul-christianos-barista. 

When I left my previous job in April of last year, I took a long break from thinking about AI safety. This was nice. Thinking about catastrophic risks all the time wears you down. Imagining oneself as Cassandra, baffled that nobody around you is taking this sufficiently seriously, is not good for the soul^[1]. I was angry at the community that had got me interested in AI risk in the first place. I felt powerless to help, and that I had earned more selfishness anyway.

Of course, avoiding AI altogether wasn’t an option. I was worried that coding would soon be automated, and my most marketable skill would disappear. I wanted to broaden my skillset and position myself well in case I couldn’t go back to a regular software engineering job. I started writing posts about how I was going to found a company, after having been heavily influenced by reading The Intelligence Curse^[2]. I got really into thinking about the intersection of AI and psychiatry. I started talking about the importance of defensive acceleration and how effective altruists systematically underrate comparative advantage. This was motivated reasoning; I wanted to work on something other than AI safety and not feel guilty about it^[3]. I figured that all these super smart, status-y types had it under control already.

                                          The EMH is trivially false across all domains

In April of this year, I had various conversations that led to me feeling excited about being part of the EA community again. I went to EAG London and primarily talked about things other than AI safety^[4]. I started doing some contract work for an EA-flavoured global health and development charity. I was managing to feel like I was part of the community without getting sucked into AI safety’s gravity well.

I ended up applying for a job at an EA organisation. I managed to get through to the work trial stage, which involved giving people advice. It was noteworthy that diverse groups of people often seemed to need to hear relatively similar advice. I kept repeating something along the lines of:

Build a model of the world. Try to falsify the components of that model. Seriously think through the implications of this model, and then act accordingly.

And then of course something deeply upsetting happened. I realised that I was not following my own advice.

So I went back to basics. Claude and I went back-and-forth over my litany of questions and objections to the idea that advanced AI systems could pose an existential threat. I wrote a post where I attempted to reason through what I thought was likely to happen over the next two years. It started to seem likely that, at the very least, things are going to get weird, soon.

Somewhere during the process it dawned on me that vanishingly few people do this kind of thing; most were LARPing, just like me. When considering something complex, at some point, it’s all too easy to let rigour lapse and hand-wave something away. Thinking rigorously is hard, but in a particularly subtle way that makes this kind of hand-waving easily done without being consciously noticeable.

I felt a sense of clarity. It might be that this clarity is entirely fake and I’m totally wrong, but still, it persists. I was ready to say it with roon:

experts are fake, smart generalists rule the world, everything is designed by people no smarter than you, courage is in shorter supply than genius

I’ve long felt like I’m trapped in a midwit meme, painfully aware of the importance of AI safety but simply lacking the intelligence or skills to actually do anything about it. When I read Scott Alexander’s Parable of the Talents, I imagined myself as Paul Christiano’s personal barista. But I don’t think any of it is true. I’ve sufficient genius. What I was lacking was the courage to actually do the hard work of thinking seriously about things for long enough to reach a conclusion, even when that conclusion is an unpleasant one.

Having sat down and made a good-faith effort to think it through, I do not think we are on track to develop AGI in a safe way. I think I have enough power and resources that there’s some chance I could make a difference. And so I’m going to try.

1. ^{^}

   Although Michael Burry seems to be making it work for him

2. ^{^}

   I feel weird about this. In some sense, it is true; I am the sole director of a company. The word “found” has various implications that aren’t accurate in my case, but sometimes it is useful to pretend that they are. Sometimes “founding” is quite high-status, and so there’s a temptation to play it up. I think that’s pretty clearly what was happening in this post. The whole thing seems naïve too – although I think that finding success with a startup probably does require some amount of naïveté.

3. ^{^}

   Despite being motivated, I do think there’s some legitimacy to the claim regardless.

^{^}

It was my sixth EAG, and the other five had all been dominated by the topic.

Show all footnotes

I wrote a case study on the School for Moral Ambition's storytelling. I came to SMA from EA, and many of its ideas sounded familiar. What surprised me was how far a new organization got with them. It seemed to reach people EA rarely reaches, with much less pushback than EA tends to get. That's an impression I can't measure.

I think SMA rewrites the usual nonprofit cast. The reader is the hero, a system is the villain, and history is the judge. Every cause comes with a named opponent, like the tobacco industry or tax rules that favor billionaires, and the reader stays off the list of suspects. Its food explainer goes after intensive animal agriculture, and veganism doesn't come up.

Is an easier version of these ideas a fair price for reaching people EA never reached? I'm unsure. Pushback welcome, and suggestions for who to look at next.

https://annapit.substack.com/p/hero-villain-judge-how-the-school?r=2fkhlo

Someone on the plane complimented my EAG NYC shirt and I mentioned it was from a conference. She took a closer look and said “oh effective altruism!” I just stared her blankly for a bit, eventually I ask “have you heard about it?” She replies “yeah it’s everywhere now, I don’t know what it means but it’s everywhere”

How times have changed.

For my SPAR project of "Hiring resources for new AI Safety orgs", this past week I've been scheduling meetings with hiring managers, and have spoken to some fantastic recruiters on the issues they are experiencing in AI Safety hiring. Here's a couple of interesting things I've learned this week:

- Screening call questions are not consistant, and feedback from canditates is that it's more a test about their alignment (and understanding) of AI Safety vs. using a common set of questions every time that screen someone that asks instead about their career goals, what they are good at, what they are not interested in doing, and what their previous peers/managers/co-workers would rate their work performance. 

- There also seems to a disconnect on how hiring managers are determining work tests for non-researcher or technical roles. This is an uncertainity that I want to explore more, as I'm unsure who is creating these work tests and who is grading them?

- A theme that keeps on coming up is the need for a "shared database" of canditates between AI Safety organizations. I heard this today from a recruiter, as a best practice from an outside field, and have a central theme on this forum for the need. 

- Most AI Safety orgs are ramping up hiring right now and include "HR" type roles (generally called generalist) that mention that they want to scale 2x in headcount this year. I had an interesting conversation with a recruiter that there is a difference in company culture, skill set, etc. for a canditate who can scale a business or team of 1-15 people vs. scaling from 20+people. When you're a smaller org, everyone is "all hands on deck", but when your organization grows, that tactic becomes unmanagable and unaligned with both what the individual contributors are focusing on, and what your organization needs. It's the understanding that it's two very different types of leaders who run those orgs. 

- I continue to build out my contact list of hiring managers and companies in AI Safety. I'm focusing on newer organizaitions - potentlally less than 10 people, and I'm looking for help with recommendations on who to speak to. Please reach out if you can help, thanks!

“We want lab employees to show up”

Can someone explain to me the over-blown importance of lab employees? That we need them to show up to our events or community or snag a conversation with them at an EAG = some sort of impact. (I.e I speculate lab employees showing up to EA SF would mean we get more funding or something) 

My understanding is that many employees who would show up are already AIS-pilled and doing their best. What more do you want from them? To leave, to unionize, to influence them in some way? 

I’m sure I’m missing something here 🤷‍♀️ 

Three big unaddressed problems in EA 2.0:

• corruption (i.e., the quid pro quo between FTX and Manifund, in which Austin Chen traded Caroline Ellison a job out of prison for cash obtained in the course of stealing customer funds)
• cults (e.g., Leverage Research and Dragon Army Barracks, whose former members/leaders such as Tyler Alterman and Duncan Sabien still post on the EA Forum)
• misinformation (e.g., METR's use of fabricated data in the famous time horizons graph)

EA sadly seems sunk to me because dire problems keep mounting and most people seem to think doing nothing is the best solution. There is also a disturbingly loud chorus of voices saying no, no, corruption is actually good (it's what having principles looks like), cults are actually good (it's part of what makes the community unique), misinformation is actually good (we're smarter than everyone else and need to save the world), which makes EA seem doomed. The force of immorality in EA is too strong and there is no equally strong force to stand against it.

EA today is at best morally ambiguous, if not more evil than good. Things have gotten really rotten. No one is doing enough to clean it up. And there's apparently no floor to how much worse it can get.

One of my fears is that there will be an EA mass shooting or terrorist attack because this almost already happened with Stop AI, an EA-funded and EA-supported organization. Sam Kirchner, the Stop AI activist who threatened to do a mass shooting at OpenAI, is still missing. There is a warrant out for his arrest.

Besides that, a disturbing number of people in the EA community seem sympathetic to far-right, illiberal, racist views (e.g., the Manifest afterparties hosted by Curtis Yarvin, who holds authoritarian and white nationalist views).

So, really, that's five big unaddressed problems in EA 2.0, including violent extremism and authoritarian, racist, far-right political views.

People who want to focus on global poverty and global health and development should probably pour their energy into things not called "EA", like effective giving. Or more mainstream humanitarian movements like Rotary International, or Hank and John Green's fundraising through things like Project for Awesome.

I still like the term "EA 1.0", but it's probably too cryptic and confusing for anyone not deeply familiar with effective altruism. "EA 1.0" could possibly be used to describe something (e.g., in an FAQ), such as an effective giving org, but it would probably be too inaccessible to use in the name.

It can be extremely painful to think about cutting your losses with EA, but we are called on in this life to make difficult decisions, to sacrifice, and to suffer when it's the right thing to do. And, in the long run, that ultimately makes you feel more whole and right inside than ethically compromising and ethically compromising... There is a kind of joy in morality.

Developing Thoughts on the Animal Advocacy Community in Africa

African animal advocacy movement may be under-exploring its potential and is NOT talent-constrained. Instead, the movement is currently held back by the following:

Lack of strategy R&D: This stems from two sources: limited exploratory, outside-the-box thinking, and funding habits that do not reward thorough research and the development of relevant interventions.

Lack of space for innovation: Few, if any, African animal advocacy organizations have a dedicated special or new projects team, because most are scrambling to keep running into the next year. Most do not dedicate enough staff and resources to developing stealth projects that could one day spin off or open a new area of work for the organization.

High risk aversion? This may be an underlying reason for the constraints above and their knock-on effects. What level of risk-taking is reasonable? Who leads or guides the norm- and behavior-setting conversations that reduce in-fighting?

Cost of recruiting and training people from outside the movement: Africa has an abundant talent pool, so African groups are not talent-constrained. Rather, groups are unable to absorb this talent for several reasons, which used to be reduced to "funding". Some may say funding is being unlocked, with the dollar amounts going to African groups growing year on year, and that the barrier now surfacing is the cost of bringing in people from this large talent pool and training them. However, this growth starts from a very low base, and much of it is short-term and restricted to specific projects rather than covering core costs such as recruiting and training staff.

Program support and MEL (monitoring, evaluation and learning): Several groups lack dedicated, experienced operational, administrative and program staff. A single person may be responsible for all three of these functions and more. As a result, proper monitoring, designated learning opportunities and communication of impact are heavily deprioritized, and in most cases non-existent.

Low wages: This is a controversial area. Most animal advocacy groups pay significantly lower salaries than other local NGOs in the same country, such as health, development and conservation NGOs. They also offer few or no benefits such as health insurance, pension or professional development. As a result, the movement loses talent to these neighboring sectors. This is in sharp contrast to international GH&D (global health and development) organizations, which have been criticized for paying high wages.

Outsider "Africa is cheap" expectation: Some people I speak with who have no experience of working in Africa assume that running projects there is so cheap that even current funding levels should have an outsized impact. Is the EA marketing of "$5,000 to save a life", based on GH&D programs in Africa, reinforcing this idea of cheapness?

Upsides

Large talent pool: The few jobs, fellowships and trainings that are announced now receive far more applications than there are places. Animal Welfare League received over 2,000 applications for a single role.

Efficiency and effectiveness: A small number of stronger groups are developing structures, projects and budgets that capture real costs and impact size, and that achieve key outcomes at lower cost. They may soon benchmark their cost-effectiveness against each other and against global groups. These groups remain the exception: most still lack the MEL capacity this requires.

New organizations: Much more can happen with new organizations, and it is exciting to see them springing up in new areas. To see what new organizations can add, compare Ghana's fish welfare space before and after SHARED (run by @Emmanuel Awuni) and AWASH (run by @Naveeth Basheer)