Does it worry you that there are very few published peer reviewed treatments of why AGI risk should be taken seriously that are relevant to current machine learning technology?

What is your solution to Pascal's Mugging?

What's a regular disagreement that you have with other researchers at FHI? What's your take on it and why do you think the other people are wrong? ;-)

We're currently in a time of global crisis, as the number of people infected by the coronavirus continues to grow exponentially in many countries. This is a bit of a hard question, but a time of crisis is often the time when governments substantially refactor things because it's finally transparent that they're not working, so can you name a feasible concrete change in the UK government (or a broader policy for any developed government) that you think would put us in a far better position for future such situations, especially future pandemics that have a much more serious chance of being an existential catastrophe?

What do you think is the biggest mistake that the EA community is currently making?

Is your view that:

(i) the main thing that matters for the long-term is whether we get to the stars

(ii) This could plausibly happen in the next few centuries

(iii) therefore the main long-termist relevance of our actions is whether we survive the next few centuries and can make it to the stars?

Or do you put some weight on the view that long-term human and post-human flourishing on Earth could also account for >1% of the total plausible potential of our actions?

What have you changed your mind on recently?

There are many ways that technological development and economic growth could potentially affect the long-term future, including:

• Hastening the development of technologies that create existential risk (see here)
• Hastening the development of technologies that mitigate existential risk (see here)
• Broadly empowering humanity (see here)
• Improving human values (see here and here)
• Reducing the chance of international armed conflict (see here)
• Improving international cooperation (see the climate change mitigation debate)
• Shifting the growth curve forward (see here)
• Hastening the colonization of the accessible universe (see here and here)

What do you think is the overall sign of economic growth? Is it different for developing and developed countries?

Note: The fifth bullet point was added after Toby recorded his answers.

If you could only convey one idea from your new book to people who are already heavily involved in longtermism, what would it be?

Can you tell us a specific insight about AI that has made you positively update on the likelihood that we can align superintelligence? And a negative one?

What are the three most interesting ideas you've heard in the last three years? (They don't have to be the most important, just the most surprising/brilliant/unexpected/etc.)

Do you think we will ever have a unified and satisfying theory of how to respond to moral uncertainty, given the huge structural and substantive differences between apparently plausible moral theories? Will MacAskill's thesis is one of the best treatments of this problem, and it seems like it would be hard to build an account of how one ought to respond to e.g. Rawlsianism, totalism, libertarianism, person-affecting views, absolutist rights-based theories, and so on, across most choice situations.

What do you think is the strongest argument against working to improve the long-term future? What do you think is the strongest argument against working to reduce existential risk?

Can you describe what you think it would look like 5 years from now if we were in a world that was making substantially good steps to deal with the existential threat of misaligned artificial general intelligence?

Should non-suffering focused altruists cooperate with suffering-focused altruists by giving more weight to suffering than they otherwise would given their worldview (or given their worldview adjusted for moral uncertainty)?

Do you think there are any actions that would obviously decrease existential risk? (I took this question from here.) If not, does this significantly reduce the expected value of work to reduce existential risk or is it just something that people have to be careful about (similar to limited feedback loops, information hazards, unilateralist's curse etc.)?

If you could convince a dozen of the world's best philosophers (who aren't already doing EA-aligned research) to work on topics of your choice, which questions would you ask them to investigate?

Can you tell us something funny that Nick Bostrom once said that made you laugh? We know he used to do standup in London...

On balance, what do you think is the probability that we are at or close to a hinge of history (either right now, this decade, or this century)?

What are the most important new ideas in your book for someone who's already been in the EA movement for quite a while?

What are your top three productivity tips?

Do you think that climate change has been neglected in the EA movement? What are some options that seem great to you at the moment to have a very large impact to stir us in a better direction regarding climate change?

We have a lot of philosophers and philosophically-minded people in EA, but only a tiny number of them are working on philosophical issues related to AI safety. Yet from my perspective as an AI safety researcher, it feels like there are some crucial questions which we need good philosophy to answer (many listed here; I'm particularly thinking about philosophy of mind and agency as applied to AI, a la Dennett). How do you think this funnel could be improved?

What's a book that you read and has impacted how you think / who you are, that you expected most people here won't have read?

Can you describe a typical day in your life with sufficient granularity that readers can have a sense of what "being a researcher at a place like FHI" is like?

What's up with Pascal's Mugging? Why hasn't this pesky problem just been authoritatively solved? (and if it has, what's the solution?) What is your preferred answer? / Which bullets do you bite (e.g., bounded utility function, assigning probability 0 to events, a decision-theoretical approach cop-out, etc.)?

Which ethical views do you have non-negligible credence in and, if true, would substantially change what you think ought to be prioritized, and how? How much credence do you have in these views?

Suppose your life's work ended up having negative impact. What is the most likely scenario under which this could happen?

What's one book that you think most EAs have not yet read and you think that they should (other than your own, of course)?

What are some of your current challenges? (maybe someone in the audience can help!)

What are you looking for in a research / operations colleague?

How robust do you think the case is for any specific longtermist intervention? E.g. do new considerations constantly affect your belief in their cost-effectiveness, and by how much?

In your book, you define an existential catastrophe as "the destruction of humanity's longterm potential". Would defining it instead as "the destruction of the vast majority of the longterm potential for value in the universe" capture the concept you wish to refer to? Would it perhaps slightly more technically accurately/explicitly capture what you wish to refer to, just perhaps in a less accessible or emotionally resonating way?

I wonder this partly because you write:

It is not that I think only humans count. Instead, it is that hu

Do you think the problems of infinite ethics give us reason to reject totalism or long-termism? If so, what is the alternative?

What are your thoughts on the argument that the track record of robustly good actions is much better than that of actions contingent on high uncertainty arguments? (See here and here at 34:38 for pushback.)

How confident are you that the solution to infinite ethics is not discounting? How confident are you that the solution to the possibility of an infinitely positive/infinitely negative world automatically taking priority is not capping the amount of value we care about at a level low enough to undermine longtermism? If you're pretty confident about both of these, do you think additional research on infinites is relatively low priority?

How much uncertainty is there in your case for existential risk? What would you put as the probability that, in 2100, the expected value of a substantial reduction in existential risk over the course of this century will be viewed by EA-minded people as highly positive? Do you think we can predict what direction future crucial considerations will point based on what direction past crucial considerations have pointed?

What do you think of applying Open Phil's outlier opportunities principle to an individual EA? Do you think that, even in the absence of instrumental considerations, an early career EA who thinks longtermism is probably correct but possibly wrong should choose a substantial chance of making a major contribution to increasing access to pain relief in the developing world over a small chance of making a major contribution to reducing GCBRs?

Is the cause area of reducing great power conflict still entirely in the research stage or is there anything that people can concretely do? (Brian Tse's EA Global talk seemed to mostly call for more research.) What do you think of greater transparency about military capabilities (click here and go to 24:13 for context) or promoting a more positive view of China (same link at 25:38 for context)? Do you think EAs should refrain from criticizing China on human rights issues (click here and search the transcript for "I noticed that over the last few ... (read more)

What are your thoughts on these questions from page 20 of the Global Priorities Institute research agenda?

How likely is it that civilisation will converge on the correct moral theory given enough time? What implications does this have for cause prioritisation in the nearer term?

How likely is it that the correct moral theory is a ‘Theory X’, a theory radically different from any yet proposed? If likely, how likely is it that civilisation will discover it, and converge on it, given enough time? While it remains unknown, how can we properly hedg

What are your views on the prioritization of extinction risks vs other longtermist interventions/causes?

Which interventions/causes do you think are best to support/work on according to views in which extra people with good or great lives not being born is not at all bad (or far outweighed by other considerations)? E.g. different person-affecting views, or the procreation asymmetry.

You seem fairly confident that we are at "the precipice", or "a uniquely important time in our story". This seems very plausible to me. But how long of a period are you imagining for the precipice?

The claim is much stronger if you mean something like a century than something like a few millennia. But even if the "hingey" period is a few millennia, then I imagine that us being somewhere in it could still be quite an important fact.

(This might be answered past chapter 1 of the book.)

Do you lean more towards a preferential account of value, a hedonistic one, or something else?

How do you think tradeoffs between pleasure and suffering are best grounded according to a hedonistic view? It seems like there's no objective one-size-fits-all trade-off rate, since it seems like you could have different people have different preferences about the same quantities of pleasure and suffering in themselves.

What new evidence would cause the biggest shifts in your priorities?

Can you describe what you think it would look like 5 years from now if we were in a world that was making substantially good steps to deal with the existential threat of engineered pandemics?

What do you like to do during your free time?

There will be a lot to learn from the current pandemic from global society. Which lesson would be most useful to "push" from EA's side?

I assume this question is in between the "best lesson to learn" and "lesson most likely to be learned". We probably want to push a lesson that's useful to learn, and that our push actually helps to bring it into policy.

What are your thoughts on how to evaluate or predict the impact of longtermist/x-risk interventions, or specifically efforts to generate and spread insights on this matters? E.g., how do you think about decisions like which medium to write in and whether to focus on generating ideas vs publicising ideas vs fundraising?

How would your views change (if at all) if you thought it was likely that there are intelligent beings elsewhere in the universe that "are responsive to moral reasons and moral argument" (quote from your book)? Or if you thought it's likely that, if humans suffer an existential catastrophe, other such beings would evolve on Earth later, with enough time to potentially colonise the stars?

Do your thoughts on these matters depend somewhat on your thoughts on moral realism vs antirealism/subjectivism?

What are some of your favourite theorems, proofs, algorithms, and data structures?

What are some directions you'd like the EA movement or some parts of the EA movement to take?

If you've read the book 'So good they can't ignore you', what do you think are the most important skills to master to be a writer/philosopher like yourself?