FHI - RA to Nick Bostrom (previously RA to Toby Ord on The Precipice)

Wiki Contributions


How many lives has the U.S. President's Emergency Plan for AIDS Relief (PEPFAR) saved?

This Vox article by Dylan Matthews cites these two studies, which try to get at this question:

EDIT to add: here's a more recent analysis, looking at mortality impact up to 2018 — Kates et al. (2021)

Ethics of existential risk

btw — there's a short section on this in my old Existential Risk wikipedia draft. maybe some useful stuff to incorporate into this. 

Ethics of existential risk

weak disagree. FWIW lots of good cites in endnotes to chapter 2 of The Precipice pp.305–12; and  Moynihan's X-Risk.

What are things everyone here should (maybe) read?

I considered writing a post about the same biography you mentioned for the forum.

I would love to read such a post! 

It's very humbling to see how much he already thought of, which we now call EA. 

Agreed — I think the Ramsey/Keynes-era Apostles would make an interesting case study of a 'proto-EA' community. 

Should EA Buy Distribution Rights for Foundational Books?

Another historical precedent

In 1820, James Mill seeks permission for a plan to print and circulate 1,000 copies of his Essay on Government, originally published as a Supplement to Napier's Encyclopaedia Britannica:

I have yet to speak to you about an application which has been made to me as to the article on Government, from certain persons, who think it calculated to disseminate very useful notions, and wish to give a stimulus to the circulation of them. Their proposal is, to print (not for sale, but gratis distribution) a thousand copies. I have refused my consent till I should learn from you, whether this would be considered an impropriety with respect to the Supplement. To me it appears the reverse, as the distribution would in some degree operate as an advertisement.

Ernest Barker suggests it was quite successful:

Mill's article was thus given a wider circulation that the Supplement to the Encyclopedia would have afforded by itself ... By 1824 ... there had appeared what was possibly a second edition ... Mill, in a letter of August 1825, speaks of the second reprint 'being all gone, and great demand remaining.' (He also mentions ... that his essays 'are the text-books of the young men of the Union at Cambridge'.

Ethics of existential risk

FWIW, and setting aside stylistic considerations for the Wiki, I dislike 'x-risk' as a term and avoid using it myself even in informal discussions. 

  • it's ambiguous between 'extinction' and 'existential', which is already a common confusion
  • it seems unserious and somewhat flippant (vaguely comic book/sci-fi vibes)
  • the 'x' prefix can denote the edgy, or sexual  (e.g. X Games; x-rated; Generation X?)
  • 'x' also often denotes an unknown value (e.g. in 'Cause X' — another abbreviation I dislike; or indeed Stefan's comment earlier in this thread)
Ethics of existential risk

I prefer this option to all others mentioned here. 

What are things everyone here should (maybe) read?

I also kind of think everyone should read at least one biography, in particular of people who have become scientifically, intellectually, culturally, or politically influential.

Some biographies I've enjoyed in this vein:

  • Frank Ramsey: A Sheer Excess of Powers
  • The Price of Peace: Money, Democracy, and the Life of John Maynard Keynes
  • Karl Marx: a Nineteenth-Century Life
Some learnings I had from forecasting in 2020

With regards to the AGI timeline, it's important to note that Metaculus' resolution criteria are quite different from a 'standard' interpretation of what would constitute AGI[1], (or human-level AI[2], superintelligence[3], transformative AI, etc.). It's also unclear what proportion of forecasters have read this fine print (interested to hear others' views on this), which further complicates interpretation.

For these purposes we will thus define "an artificial general intelligence" as a single unified software system that can satisfy the following criteria, all easily completable by a typical college-educated human.

  • Able to reliably pass a Turing test of the type that would win the Loebner Silver Prize.
  • Able to score 90% or more on a robust version of the Winograd Schema Challenge, e.g. the "Winogrande" challenge or comparable data set for which human performance is at 90+%
  • Be able to score 75th percentile (as compared to the corresponding year's human students; this was a score of 600 in 2016) on all the full mathematics section of a circa-2015-2020 standard SAT exam, using just images of the exam pages and having less than ten SAT exams as part of the training data. (Training on other corpuses of math problems is fair game as long as they are arguably distinct from SAT exams.)
  • Be able to learn the classic Atari game "Montezuma's revenge" (based on just visual inputs and standard controls) and explore all 24 rooms based on the equivalent of less than 100 hours of real-time play (see closely-related question.)

By "unified" we mean that the system is integrated enough that it can, for example, explain its reasoning on an SAT problem or Winograd schema question, or verbally report its progress and identify objects during videogame play. (This is not really meant to be an additional capability of "introspection" so much as a provision that the system not simply be cobbled together as a set of sub-systems specialized to tasks like the above, but rather a single system applicable to many problems.)

  1. OpenAI Charter ↩︎

  2. expert survey ↩︎

  3. Bostrom ↩︎

Has anyone gone into the 'High-Impact PA' path?

I work at FHI, as RA and project manager for Toby Ord/The Precipice (2018–20), and more recently as RA to Nick Bostrom (2020–). Prior to this, I spent 2 years in finance, where my role was effectively that of an RA (researching cement companies, rather than existential risk). All of the below is in reference to my time working with Toby.

Let me know if a longer post on being an RA would be useful, as this might motivate me to write it.


I think a lot of the impact can be captured in terms of being a multiplier[1] on their time, as discussed by Caroline and Tanya. This can be sub-divided into two (fuzzy, non-exhaustive) pathways:

  • Decision-making — helping them make better decisions, ranging from small (e.g. should they appear on podcast X) to big (e.g. should they write a book)
  • Execution — helping them better execute their plans

When I joined Toby on The Precipice, a large proportion of his impact was ‘locked in’ insofar he was definitely writing the book. There were some important decisions, but I expect more of my impact was via execution, which influenced (1) the quality of the book itself; (2) the likelihood of it’s being published on schedule; (3) the promotion of the book and its ideas; (4) the proportion of Toby’s productive time it took up, i.e. by freeing up time for him to do non-book things. Over the course of my role, I think I (very roughly) added 5–25% to the book’s impact, and freed up 10–33% of Toby's time.

Career decisions

Before joining Toby, I was planning to join the first cohort of FHI’s Research Scholars Program and pursued my own independent projects for 2 years. At the time, the most compelling reason for choosing the RA role was:

  • Toby’s book will have large impact X, and I can expect to multiply this by ~10%, for impact of ~0.1X
  • If I ‘do my own thing’, it would take me much longer than 2 years to find and execute a project with at least 0.1X impact (relative to the canonical book on existential risk…)

One thing I didn’t foresee is how valuable the role would be for my development as a researcher. While I’ve had less opportunity to choose my own research projects; publish papers; etc., I think this has been substantially outweighed by the learning benefits of working so closely with a top tier researcher on important projects. Overall, I expect that working with Toby ‘sped up’ my development by a few years relative to doing independent research of some sort.

One noteworthy feature of being a ‘high-impact RA/PA/etc’ is that while these jobs are relatively highly regarded in EA circles, they can sound a bit baffling to anyone else. As such, I think I’ve built up some pretty EA-specific career capital.

Some other names

Here's an incomplete list of people who have done (or are doing) this line of work, other than Caroline and myself:

Nick Bostrom — Kyle Scott, Tanya Singh, Andrew Snyder-Beattie

Toby Ord — Andrew Snyder-Beattie, Joe Carlsmith

Will MacAskill – Pablo Stafforini, Laura Pomarius, Luisa Rodriguez, Frankie Andersen-Wood, Aron Vallinder

  1. some RA trivia — Richard Kahn, the economist normally credited with the idea of a (fiscal) multiplier, was a long-time RA to John Maynard Keynes, of whom Keynes’ wrote “He is a marvelous critic and suggester and improver … There never was anyone in the history of the world to whom it was so helpful to submit one’s stuff.” ↩︎

Load More