Another complication here is that a lot of arguments are arguments about the expected value of some variable --- ie, the argument that we should take some action is implicitly an argument that the expected utility from taking that action is greater than that from taking the action we would have taken otherwise. 

And it's not clear what a % credence means when it comes to an estimate of an expected value --- expected values aren't random variables. Ie, if I think we ought to work on AI-risk over Global Public Health  since I think there is a 1% chance of an AI intervention saving trillions of lives, it's not clear what it'd mean to put another % confidence over that already probabilistically derived expected utility: I've already incorporated the 99% chance of failure into my case for working on AI-risk. Certainly it's good to acknowledge that chance of failure, but it doesn't say anything about my epistemic status in my argument. 

I think reporting % credences serve a purpose more similar to reporting effect sizes than an epistemic status. They're something for you to average together to get a quick & dirty estimate of what the consensus is. 

Anyway, re: what to do in the case when the argument is about an expected value --- I think the best practice is to to point out the known unknowns that you think are the most likely ways your argument might be shown to be false -- ie, "I think we should work on AI over Global Public Health, but I think my case depends on fast takeoff being true, I'm only 60% confident that it is, and I think we can get better info about which takeoff scenario is the more likely to happen."

In the case the biggest known unknowns are what priors you should have before seeing a piece of evidence, this basically reduces down to your strength of evidence/epistemic shift proposal. But I think generally when we're talking about our epistemic status, it's more useful to concentrate on how our beliefs might be changed in the future, and how qualitatively we think other people might accomplish changing our minds, than how they've changed in the past.

(It seems the correct "Bayesian" thing to do the above if you really wanted to report your beliefs using numbers would be to take your priors about information you'll receive at each time $t$ in the future, encode the structure of your uncertainty about what you'll know at each point in time as a filtration of your event space $F_t\subseteq F$, and then report your uncertainty about the trajectory about your future beliefs about $X$ as the martingale process $Y_t=E\left[X|F_t\right]$. 

Needless to say this is a pretty unwieldy and impractical way to report your epistemic status). 

I want to flesh out my impressions about what the crucial considerations for strong longtermist EA priorities are. I think I may want to put some of these into a fuller writeup eventually. I think there’s still a pressing need to evaluate the stronger claims about the value of AI alignment research more, so I wanted to put out my thoughts below about why I currently think the value of a pretty wide range of what I call medium-term interventions rest on those strong claims, especially as I'm pretty personally interested in a lot of more medium-term interventions. 

• I think there’s a rough consensus among longtermists that (a) we currently live in a time of perils due to anthropogenic x-risk and that (b) most of the moral value of humanity lies in scenarios where we escape the time of perils and that (c) we need to get out of the time of perils by developing and implementing “safety technologies” (broadly construed).
• I think that a broad range of potential interventions can be characterized as aiming to broadly increase the ability of humanity to act in the medium term future. I think these include:
  • Most interventions under the label of “patient philanthropy:”
    • Global priorities research
    • Movement building
    • Saving to give later
  • Many broad interventions currently considered lower priorities:
    • Various forms of governance and institutional reform.
    • Building infrastructure to speed up scientific progress and economic growth.
• I think a crucial consideration driving the value of these medium-term interventions relative to more targeted attempts at achieving differential technological progress is whether risk-reductions from safety technologies are diffuse or concentrated.
  • If we can identify a small number of technologies that have very large expected x-risk reductions, then it makes sense to try to identify them and begin work on them immediately.
  • On the other hand, if possible x-risk reduction is spread out over a large number of potential safety technologies, then picking out technologies to start working on now becomes a much more difficult and less fruitful task. We are likely better off with broad interventions empowering future people with more information and resources to work on these problems, thus medium-term interventions become much more valuable.
• I think our default view should be in favor of x-risk reduction being fairly diffuse across safety technologies.
• I think that the case for x-risk reduction being concentrated largely rests on what I want to call the “strong case for technical AGI-alignment research:” (a) the existential risk posed by misaligned AGI is very large and that (b) we can start research alignment technologies now that can permanently eliminate a large portion of that risk.
  • Eg, if keeping AGI risk low requires constantly coming up with new safety solutions for a long time, then broadly improving our medium-term capabilities might be a lot more useful than going all in on AGI safety research now.
  • I think we do not currently believe that there’s a small number of discrete safety technologies that can eliminate a large portion of biorisk.
• I think that this means that the case for medium-term interventions largely rests on whether or not the strong claim for technical AGI-alignment research is true. I think this means is that if we think the strong case is largely untrue, we should probably be prioritizing
• I am generally skeptical of the strong case, but I think there are a lot of uncertainties that could change my mind. I think the main factors affecting my beliefs about the strong case right now are (1) I think the strong case probably requires the Bostrom-Yudkowsky model of AGI development being true (2) I think that the Bostrom-Yudkowsky model is probably false and (3) I think that if the Bostrom-Yudkowsky model were true, technical alignment is probably largely intractable. I don’t have a huge level of confidence in any of these three claims though. (I think I want to flesh out this last point a lot more later).

Hi!

My main personal project for the summer is trying to figure out what I think about AI-risk, so I thought I should engage with the forum more to ask questions/solicit feedback. I'm currently a mathematics undergrad, about to start my 4th year, so part of this is trying to figure out whether or not I should pivot toward working in something closer to AI-risk. 

About me -- I first got interested in EA after reading Reasons and Persons in the summer of 2020. My main secondary academic interest in undergrad has been in political theory, so I'm very interested in questions such as whether naïve utilitarianism endorses political extremism, how that might be mitigated by a proper social epistemology, and what that might entail for consequentialists interested in voting/political process reform. I'm also very interested in the economics of cities and innovation, as well as understanding how we learn mathematics. I'm less sure how those topics fit in an EA framework, but I'm always interested in seeing what insights others might be able to bring to them from an EA standpoint. 

Here's hoping to learning a lot from y'all's!

-- Edgar