Rethinking the Value of Working on AI Safety

JohanEA

Rethinking the Value of Working on AI Safety

JohanEA

12 min readJan 9, 2025

Comments 21

Sorted by

New & upvoted

Jim Buhler

Do you think that AI safety is i) at least a bit good in expectation (but like with a determinate credence barely higher than 50% because high risk/uncertainty) or ii) you don't have determinate credences and feel clueless/agnostic about this? I feel like your post implicitly keeps jumping back and forth between these two positions, and only (i) could support your conclusions. If we assume (ii), everything falls apart. There's no reason to support a cause X (or the exact opposite of X) to any degree if one is totally clueless about whether it is good.

Thanks for writing this :)

JohanEA

One of the reasons I wrote this post was to reflect on excellent comments like yours. Thank you for posting and spotting this inconsistency!

You rightly point out that I jump between i) and ii). The short answer is that, at least for AI safety, I feel clueless or agnostic about whether this cause is positive in expectation. @Mo Putera summarised this nicely in their comment. I am happy to expand on the reasons as to why I think that.

What is your perspective here? If you do have a determinate credence above 50% for AI safety work, how do you arrive at this conclusion? I know you have been also doing some in-depth thinking on the topic of cluelessness.

Next, I want to push back on your claim that if ii) is correct, everything collapses. I agree that this would lead to the conclusion that we are probably entirely clueless about longtermist causes, probably the vast majority of causes in the world. However, it would make me lean toward near-term areas with much shorter causal chains, where there is a smaller margin of error—for example, caring for your family or local animals, which carry a low risk of backfiring.

Although, to be fair, this is unclear as well if one is also clueless about different moral frameworks. For example, helping a young child who fell off their skateboard might seem altruistic but could inadvertently increase their ambition, leading them to become the next Hitler or a power-seeking tech CEO. And to take this to the next level: not taking an action also has downsides (e.g not addressing the ongoing suffering in the world). Yaay!

If conclusion ii) is correct for all causes, altruism would indeed seem not possible from a consequentialist perspective. I don’t have a counterargument at the moment.

I would love to hear your thoughts on this!

Thank you for engaging :)

Jim Buhler

If you do have a determinate credence above 50% for AI safety work, how do you arrive at this conclusion?

It happens that I do not. But I would if I believed there was evidence robust to unknown unknowns in favor of assuming "AI Safety work" is good, factoring in all the possible consequences from now until the end of time. This would require robust reasons to believe that current AI safety work actually increases rather than decreases safety overall AND that increased safety is actually good all things considered (e.g., that human disempowerment is actually bad overall). (See Guillaume's comment on the distinction). I won't elaborate on what would count as "evidence robust to unknown unknowns" in such a context but this is a topic for a future post/paper, hopefully.

Next, I want to push back on your claim that if ii) is correct, everything collapses. I agree that this would lead to the conclusion that we are probably entirely clueless about longtermist causes, probably the vast majority of causes in the world. However, it would make me lean toward near-term areas with much shorter causal chains, where there is a smaller margin of error—for example, caring for your family or local animals, which carry a low risk of backfiring.

Sorry, I didn't mean to argue against that. I just meant that work you are clueless about (e.g. maybe AI safety work in your case?) shouldn't be given any weight in your diversified portfolio. I didn't mean to make any claim about what I personnally think we should or shouldn't be clueless about. The "everything falls apart" was unclear and probably unwarranted.

JohanEA

I agree with your reasoning, and the way you’ve articulated it is very compelling to me! It seems that the bar this evidence would need to reach is, quite literally, impossible.

I would even take this further and argue that your chain of reasoning could be applied to most causes (perhaps even all?), which seems valid.

Would you disagree with this?

Your reply also raises a broader question for me: What criteria must an intervention meet for our determinance credence in its expected value being positive to exceed 50%, thereby justifying work on it?

Jim Buhler

I would even take this further and argue that your chain of reasoning could be applied to most causes (perhaps even all?), which seems valid.
Would you disagree with this?

I mean, I didn't actually give any argument for why I don't believe AI safety is good overall (assuming pure longtermism, i.e., taking into account everything from now until the end of time). I just said that I would believe it if there was evidence robust to unknown unknowns. (I haven't argued that there wasn't such evidence already; although the burden of the proof is very much on the opposite claim tbf). But I think this criterion applies to all causes where unknown unknowns are substantial, and I believe this is all of them as long as we're evaluating them from a pure longtermist perspective, yes. And whether there is any cause that meets this criterion depends on one's values I think. From a classical utilitarian perspective (and assuming the trade-offs between suffering and pleasure that most longtermists endorse), for example, I think there's very plausibly none that does meet this criterion.

Guillaume Corlouer

Thanks for writing this post! It is important to think about the implications of cluelessness and moral uncertainty for AI safety. To clarify the value of working on AI safety, it helps to decompose the problem into two subquestions:

Is the outcome that we are aiming for robust to cluelessness, and moral uncertainty?
Do we know of an intervention that is robustly good to achieve that outcome? (i.e. an intervention that is at least better than doing nothing to achieve that outcome)

An outcome could be reducing X-risks from AI which could at least happen in two different ways: value lock-in from a human-aligned AI or from a non-human AI controlled future. Reducing value lock-in seems robustly good, and i won't argue for that here.

If the outcome we are thinking about is reducing extinction from AI, then the near-term case from reducing extinction from AI seems more robust to cluelessness, and I feel that the post could have emphasised it a bit more. Indeed, reducing the risk of extinction from AI, for all the people alive today and in the next few generations, looks good from a range of moral perspectives (it is at least determinate good for humans) even though it is indeterminate in the long-term. But then, one has to compare short term AI X-risks with other interventions that makes things determinately good in the short term from an impartial perspective, like working on animal welfare, or reducing extreme poverty.

AI seems high stakes, even though we don't know which way it will go in the short/long term, which might suggest focusing more on capacity building instead of a more direct intervention (I would put capacity building in some of the paths that you suggested, as a more general category to put careers like earning to give). This could hold as long as the capacity building (for ex. putting ourselves in a position to make things go well w.r.t AI when we have more information) has low risk of backfiring (don't make things worse) that is.

If we grant that reducing X-risks from AI seems robustly good, and better than alternative short-term causes (which is a higher bar than ``better than doing nothing''), then we still need to figure out interventions that are robust to reduce X-risks from AI (i.e. so that we don't make things worse). I already mentioned some non-backfire capacity building (if we find such kind of capacity building). Beyond capacity building; it's not completely clear to me that there are robustly good interventions in AI safety, and I think more work is needed to prioritize interventions.

It seems useful to think of one's career as being part of a portfolio, and work on things where one could plausibly be in a position to do excellent work, unless the intervention that one is working on is not determinately better than doing nothing.

Greg_Colbourn ⏸️

Beyond capacity building; it's not completely clear to me that there are robustly good interventions in AI safety, and I think more work is needed to prioritize interventions.

I think it's pretty clear^[1] that stopping further AI development (or Pausing) is a robustly good intervention in AI Safety (reducing AI x-risk).

^{^}
But see this post for some detailed reasoning.

JohanEA

I enjoyed reading your insightful reply! Thanks for sharing, Guillaume. You don’t make any arguments I strongly disagree with, and you’ve added many thoughtful suggestions with caveats. The distinction you make between the two sub-questions is useful.

I am curious, though, about what makes you view capacity building (CB) in a more positive light compared to other interventions within AI safety. As you point out, CB also has the potential to backfire. I would even argue that the downside risk of CB might be higher than that of other interventions because it increases the number of people taking the issue seriously and taking proactive action—often with limited information.

For example, while I admire many of the people working at PauseAI, I believe there are quite a few worlds in which those initially involved in setting up the group have had a net-negative impact in expectation. Even early on, there were indications that some people were okay with using violence or radical methods to stop AI (which was then banned by the organizers). However, what happens if these tendencies resurface when “shit hits the fan”? To push back on my own thinking, it still might be a good idea to work on PauseAI due to community diversification argument within AI safety (footnote two).

I agree that other forms of CB, such as MATS, seem more robust. But even here, I can always find compelling arguments for why I should be clueless about the expected value. For instance, an increased number of AI safety researchers working on solving an alignment problem that might ultimately be unsolvable could create a false sense of security.

Greg_Colbourn ⏸️

However, what happens if these tendencies resurface when “shit hits the fan”?

I don't think this could be pinned on PauseAI, when at no point has PauseAI advocated or condoned violence. Many (basically all?) political campaigns attract radical fringes. Non-violent moderates aren't responsible for them.

Guillaume Corlouer

I am curious, though, about what makes you view capacity building (CB) in a more positive light compared to other interventions within AI safety. As you point out, CB also has the potential to backfire. I would even argue that the downside risk of CB might be higher than that of other interventions because it increases the number of people taking the issue seriously and taking proactive action—often with limited information.

Yeah, just to clarify, CB is not necessarily better than other interventions. However, CB with low backfire risks could be promising. This does not necessarily mean doing community building, since community building could backfire depending on how it is done (for example maybe if it is done in a very expansive non-careful way it could more easily backfire). I think the PauseAI example that you gave is a good example of potentially non robust intervention, or at least I would not count it as a low backfire risk capacity building intervention.

One of the motivation of CB would be to put ourselves in a better position to pursue some intervention if we end up less clueless. It might be that we don't in fact end up less clueless, and that while we have done CB, there are still no robust interventions that we can pursue after some time. In that case, it would be better to pursue determinately good short-term interventions even after doing CB (but then we have to pay the opportunity cost of the resources spent doing CB rather than doing the interventions good in the short term directly).

I am still uncertain about low backfire CB interventions (that are better than doing something good directly), perhaps some way of increasing capital or well targeted community building could be good examples, but it seems like an open question to me.

Jan Wehner🔸

Great post Johan! It got me thinking more deeply about the value of working on x-risk reduction and how we ought to act under uncertainty. I think people (including me) doing direct work on x-risk reduction would do well to reflect on the possibility of their work having (large) negative effects.

I read the post as making 2 main arguments:
1) The value of working on x-risk reduction is highly uncertain
2) Given uncertainty about the value of cause areas, we should use worldview diversification as a decision strategy

I agree with 1, but am not convinced by 2). WDS might make sense for a large funder like OpenPhil, but not for individuals seeking to maximize their positive impact on the world. Diversification makes sense to reduce downside risk or because of diminishing returns of investing in one option. I think neither of those apply to individuals interested in maximizing the EV of their positive impact on the world. 1) While protecting against downside risks (e.g. losing all your money) makes sense in investing, it is not important if you're only maximizing Expected Value. 2) Each individual's contributions to a cause area won't change things massively, so it seems implausible that there are strong diminishing returns to their contributions.

However, I am in favor of your suggestion of doing worldview diversification at the community level, not at the level of individuals. To an extent, EA already does that by emphasising Neglectedness. Perhaps practically EAs should put more weight on neglectedness, rather than working on the single most important problem they see.

Mo Putera

I think you're not quite engaging with Johan's argument for the necessity of worldview diversification if you assume it's primarily about risk reduction or or diminishing returns. My reading of their key point is that we don't just have uncertainty about outcomes (risk), but uncertainty about the moral frameworks by which we evaluate those outcomes (moral uncertainty), combined with deep uncertainty about long-term consequences (complex cluelessness), leading to fundamental uncertainty in our ability to calculate expected value at all (even if we hypothetically want to as EV-maximisers, itself a perilous strategy), and it's these factors that make them think worldview diversification can be the right approach even at the individual level.

JohanEA

Mo, thank you for chiming in. Yes, you understood the key point, and you summarised it very well! In my reply to Jan, I expanded on your point about why I think calculating the expected value is not possible for AI safety. Feel free to check it out.

I am curious, though: do you disagree with the idea that a worldview diversification approach at an individual level is the preferred strategy? You understood my point, but how true do you think it is?

JohanEA

Hi Jan, I appreciate the kind words and the engagement!

You correctly summarized the two main arguments. I will start my answer by making sure we are on the same page regarding what expected value is.

Here is the formula I am using:

(EV) = (p) × (+V) + (1 − p) × (−W)
p = probability
V = magnitude of the good outcome
−W = magnitude of the bad outcome

As EAs, we are trying to maximize EV.

Given that I believe we are extremely morally uncertain about most causes, here is the problem I have encountered: Even if we could reliably estimate the probability of how good or bad a certain outcome is, and even how large +V and −W are, we still don’t know how to evaluate the overall intervention due to moral uncertainty.

For example, while the Risk-Averse Welfarist Consequentialism worldview would “aim to increase the welfare of individuals, human or otherwise, without taking long-shot bets or risking causing bad outcomes,” the Total Welfarist Consequentialism worldview would aim to maximize the welfare of all individuals, present or future, human or otherwise.

In other words, the two worldviews would interpret the formula (even if we perfectly knew the value of each variable) quite differently.

Which one is true? I don’t know. And this does not even take into account other worldviews, such as Egalitarianism, Nietzscheanism, or Kantianism, and others that exist.

To make matters worse, we are not only morally uncertain, but I am also saying that we can’t reliably estimate the probability of how likely a certain +V or −W is to come into existence through the objectives of AI safety. This is linked to my discussion with Jim about determinate credences (since I didn’t initially understand this concept well, ChatGPT gave me a useful explanation).

I think complex cluelessness leads to the expected value formula breaking down (for AI safety and most other longtermist causes) because we simply don’t know what p is, and I, at least, don’t have a determinate credence higher than 50%.

Even if we were to place a bet on a certain worldview (like Total Welfarist Consequentialism), this wouldn’t solve the problem of complex cluelessness or our inability to determine p in the context of AI safety (in my opinion).

This suggests that this cause shouldn’t be given any weight in my portfolio and implies that even specializing in AI safety on a community level doesn’t make sense.

Having said that, there are quite likely some causes where we can have a determinate credence way above 50% for p in the EV formula. And in these cases, a worldview diversification strategy seems to be the best option. This requires making a bet on a set or one worldview, though. Otherwise, we can’t interpret the EV formula, and doing good might not be possible.

Here is an (imperfect) example of how this might look and why a WDS could be the best to pursue. The short answer to why WDS is preferred appears to be that, given moral uncertainty, we don't need to choose one out of 10 different worldviews and hope that it is correct; instead, we can diversify across different ones. Hence, this seems to be a much more robust approach to doing good.

What do you make of this?

Anthony DiGiovanni 🔸

This is linked to my discussion with Jim about determinate credences (since I didn’t initially understand this concept well, ChatGPT gave me a useful explanation).

FYI, I don't think ChatGPT's answer here is accurate. I'd recommend this post if you're interested in (in)determinate credences.

Greg_Colbourn ⏸️

You mention S-risk. I tend to try not to think too much about this, but it needs to be considered in any EV estimate of working on AI Safety. I think appropriately factoring it in could be overwhelming in terms of concluding that preventing ASI from being built is the number 1 priority. Preventing space colonisation x-risk could be achieved by letting ASI extinct everything. But how likely is ASI-induced extinction over ASI-induced S-risk (ASI simulating, or physically creating, astronomical amounts of unimaginable suffering, on a scale far larger than human space colonisation could ever achieve)?

Deogin

Just flagging, it seems pretty strange to have something about career choice in 'Community'

JohanEA

Thank you for flagging. I actually tried removing the tag before you mentioned this, since I agree. I tried removing it multiple times already (when I wanted to replace it with a better tag), but it isn't working ...

Just before I posted it, I selected the tag because it stated: "The community topic covers posts about the effective altruism community, as well as applying EA in one's personal life". But the subsequent points didn't match too well, and I didn't think that it would be shown in a separate section.

@Toby Tremlett🔹 or @Sarah Cheng , could you please remove the tag?

Greg_Colbourn ⏸️

Going to push back a bit on the idea that preventing AI x-risk might be net negative as a reason for not trying to do this. Applying the reversal test: given the magnitude of expected values either way, positive or negative, from x-risk reduction, it would be an extreme coincidence if doing nothing was the best action. Why not work to increase x-risk? I appreciate there are other second order effects that may be overwhelming - the bad optics of being viewed as a "supervillain" for one! But this perhaps is a useful intuition pump for why in fact decreasing x-risk is likely to actually be +EV.

Greg_Colbourn ⏸️

-4

I'm concerned that given the nearness of AI-x-risk and high likelihood that we all die in the near future, to some extent you are trying to seek comfort in complex cluelessness and moral uncertainty. If we go extinct (along with all the rest of known life in the universe), maybe it would be for the best? I don't know, I think I would rather live to find out, and help steer the future toward more positive paths (we can end factory farming before space colonisation happens in earnest). I also kind of think "what's the point in doing all these other EA interventions if the world just ends in a few years?" Sure, there is some near term benefit to those helped here and now, but everyone still all ends up dead.

Greg_Colbourn ⏸️

To be clear, given the situation we now find ourselves in, I also think it's perfectly reasonable to work on AI x-risk reduction for selfish reasons. Or just wanting your family and friends to live.

Comments

JohanEA

Hi Jan, I appreciate the kind words and the engagement!

You correctly summarized the two main arguments. I will start my answer by making sure we are on the same page regarding what expected value is.

Here is the formula I am using:

(EV) = (p) × (+V) + (1 − p) × (−W)
p = probability
V = magnitude of the good outcome
−W = magnitude of the bad outcome

As EAs, we are trying to maximize EV.

In other words, the two worldviews would interpret the formula (even if we perfectly knew the value of each variable) quite differently.

Which one is true? I don’t know. And this does not even take into account other worldviews, such as Egalitarianism, Nietzscheanism, or Kantianism, and others that exist.

This suggests that this cause shouldn’t be given any weight in my portfolio and implies that even specializing in AI safety on a community level doesn’t make sense.

What do you make of this?

^{^}

A general note for people still choosing their career path:

If you accept the arguments of moral uncertainty and complex cluelessness, and you recognize that the overall importance of AI safety remains unclear after considering these points, relying solely on the community diversification strategy to justify continuing work in AI safety becomes less compelling. The "importance" domain should carry less weight in your decision-making, while neglectedness and personal fit should be given greater emphasis. Assuming an equal level of personal fit, this could lead you to favor pivoting toward areas that are more neglected or where progress is easier to achieve. Within the EA community, AI safety already receives substantial attention and resources.

^{^}

However, the argument for diversification at a broader community level also applies within AI safety, offering greater assurance that your work contributes to the overall diversification of the AI safety community, even if you can't be sure about how good the specific thing is you are working on.

^{^}

The development of AGI will likely introduce many new crucial new considerations that render current diversification efforts within the EA community less effective. In such scenarios, you could have a greater impact by funding neglected and important work or alternative approaches to a cause.

^{^}

A counterargument to this reasoning is that it is highly uncertain whether you would be able to significantly influence society post-AGI, even with substantial capital. Speculating about the state of affairs once AGI is developed is inherently difficult. Furthermore, if we succeed in building aligned ASI, it is unclear to what extent humans would be needed to solve global problems, as ASI might handle these tasks far more effectively.

Rethinking the Value of Working on AI Safety

Rethinking the Value of Working on AI Safety

Summary

Introduction

1. Complex Cluelessness

2. Moral Uncertainty

How a worldview diversification strategy can address this issue

What are the practical implications?

Which of these paths should you pursue?

Meta Career

Pros

Cons

Further Reading

Specialising in a specific cause (using AI safety as an example^[1])

Pros

Cons

When you should consider choosing / continuing this path:

Earning to give

Pros

Cons

When you should consider choosing this path:

An important consideration

Final Reflections

Invitation to a Discussion Session

Acknowledgements

Rethinking the Value of Working on AI Safety

Rethinking the Value of Working on AI Safety

Summary

Introduction

1. Complex Cluelessness

2. Moral Uncertainty

How a worldview diversification strategy can address this issue

What are the practical implications?

Which of these paths should you pursue?

Meta Career

Pros

Cons

Further Reading

Specialising in a specific cause (using AI safety as an example[1])

Pros

Cons

When you should consider choosing / continuing this path:

Earning to give

Pros

Cons

When you should consider choosing this path:

An important consideration

Final Reflections

Invitation to a Discussion Session

Acknowledgements

Specialising in a specific cause (using AI safety as an example^[1])