Due to 2, I think I might go as high as 0.1% risk over the next 20 years (with incredibly wide error bars). That still seems high enough to seriously consider, although I agree other X-risk concerns are likely even higher.
This post feels like it is saying that lone actors are currently incapable of creating an X-risk bioweapon. I agree. I think almost everyone in the X-risk scene agrees.
The concern is instead about whether this capability will exist soon. To show that being impossible (or very unlikely), IMO you need to show that there is at least one critical step that is not getting democratized. IE, bio-expertise is getting cheaper because of increasingly powerful open source models. Robots with sufficient dexterity to do bio experiments are getting more capable and cheaper. Where is the bottleneck that will remain unsolved for 2+ decades, and why are we confident it stays unsolved?
To give an analogy, 2 years ago non-experts couldn't solve major open math problems. Today, with access to frontier models, they can. How can we be confident that the bottlenecks against independently creating novel & powerful plagues will not also fall?
Reading that section does not change my opinion on this at all, fwiw.
Maybe this will help convey my intuition. Imagine
A100*: As A100, but every shock is accompanied by a man saying a different random word.
Now, the experience of each shock in A100* is different. So A100* is not 100X as bad as A1. But I think most people would agree that A100 and A100* are not meaningfully different.
I think your notion of 'subjectively identical experiences' is not a natural way to count. And using a boolean criterion for this (identical experiences are given no extra weight, minorly different experiences are given full weight) feels awkward.
I guess I'm not sure how you're going to figure out that the subjective experience of multiple copies is identical. Even if you did, I would personally view experiencing identical joy twice (literally indistinguishable, but happened 2 times) as twice as valuable.
I think you're saying that each subjectively indistinguishable experience class only counts as 1, regardless of the number of instantiations of that experience. I can understand that, but I definitely disagree (I count the number of instantiations).
Im very unsure how we'd settle this disagreement. Maybe the following thought experiment would persuade you:
A1: You are put in a chair, and given a harmless (but painful) electric shock. Your memory of this experience is then wiped, and you are returned to your state before the shock.
A100: As above, but we repeat the procedure 100 times. Each experience is subjectively identical.
IMO A100 is worse than A1. My naive expectation is that you are neutral between the two.
My intuition is that if N copies experience something, that experience gets multiplied by N. I suspect this is closer to most people's intuition.
To give an example, 10k chickens being debeaked is about 10k as bad as one chicken being debeaked. Individual chickens are pretty close to copies from a conscious perspective (not completely so, but much closer to copies than two randomly chosen humans).
I think the cringe reason is why I'll say I'm EA, and then describe it as "quote unquote effective altruism" (if they ask). Which solves the problem decently well, in the same way folks describe themselves as 'rats' instead of rationalists.
Thanks for writing this up! I really appreciate effort-posts trying to show the efficacy of proposals, all the more so when the conclusion is negative. Thanks for pushing back against the file-drawer effect.
Oh thanks for adding concrete figures, that is very helpful (I should have done the same)!
This post feels like it is saying that lone actors are currently incapable of creating an X-risk bioweapon. I agree. I think almost everyone in the X-risk scene agrees.
The concern is instead about whether this capability will exist soon. To show that being impossible (or very unlikely), IMO you need to show that there is at least one critical step that is not getting democratized. IE, bio-expertise is getting cheaper because of increasingly powerful open source models. Robots with sufficient dexterity to do bio experiments are getting more capable and cheaper. Where is the bottleneck that will remain unsolved for 2+ decades, and why are we confident it stays unsolved?
To give an analogy, 2 years ago non-experts couldn't solve major open math problems. Today, with access to frontier models, they can. How can we be confident that the bottlenecks against independently creating novel & powerful plagues will not also fall?
Reading that section does not change my opinion on this at all, fwiw.
Maybe this will help convey my intuition. Imagine A100*: As A100, but every shock is accompanied by a man saying a different random word.
Now, the experience of each shock in A100* is different. So A100* is not 100X as bad as A1. But I think most people would agree that A100 and A100* are not meaningfully different.
I think your notion of 'subjectively identical experiences' is not a natural way to count. And using a boolean criterion for this (identical experiences are given no extra weight, minorly different experiences are given full weight) feels awkward.
Well, biting that bullet at least keeps you consistent.
Fwiw, I think my take is more common. I asked a few folks, everyone asked agreed A1 is less bad than A100.
Not saying that is strong evidence that you're wrong, but it does call for more justification of your non-standard choice.
I guess I'm not sure how you're going to figure out that the subjective experience of multiple copies is identical. Even if you did, I would personally view experiencing identical joy twice (literally indistinguishable, but happened 2 times) as twice as valuable.
I think you're saying that each subjectively indistinguishable experience class only counts as 1, regardless of the number of instantiations of that experience. I can understand that, but I definitely disagree (I count the number of instantiations).
Im very unsure how we'd settle this disagreement. Maybe the following thought experiment would persuade you:
A1: You are put in a chair, and given a harmless (but painful) electric shock. Your memory of this experience is then wiped, and you are returned to your state before the shock.
A100: As above, but we repeat the procedure 100 times. Each experience is subjectively identical.
IMO A100 is worse than A1. My naive expectation is that you are neutral between the two.
Do you mean it is bad for one's own mental health (strong agree) or that it should be discouraged (unsure if I agree)?
My intuition is that if N copies experience something, that experience gets multiplied by N. I suspect this is closer to most people's intuition.
To give an example, 10k chickens being debeaked is about 10k as bad as one chicken being debeaked. Individual chickens are pretty close to copies from a conscious perspective (not completely so, but much closer to copies than two randomly chosen humans).
I think that this post is wrong for the same reason that eigenism is wrong. Bentham's bulldog gives a better rebuttal than I would: https://benthams.substack.com/p/dan-hendrycks-moral-theory-is-very
I think the cringe reason is why I'll say I'm EA, and then describe it as "quote unquote effective altruism" (if they ask). Which solves the problem decently well, in the same way folks describe themselves as 'rats' instead of rationalists.
There is a lot already written on this topic that you may want to read.
https://forum.effectivealtruism.org/s/i9yRZjszhFf4DmQHx https://forum.effectivealtruism.org/topics/interpersonal-comparisons-of-wellbeing https://forum.effectivealtruism.org/posts/64fEWW2Zn4xK22Qy2/puzzles-for-some https://forum.effectivealtruism.org/posts/Z4otXYNjWWFHerwdw/against-population-ethics https://forum.effectivealtruism.org/posts/BFDojdoTcBuqZyCp3/utilitarianism-with-and-without-expected-utility
And I think Holden's post is a good summary for why I (and likely many others on the forum) disagree with this post. https://forum.effectivealtruism.org/posts/iupkbiubpzDDGRpka/other-centered-ethics-and-harsanyi-s-aggregation-theorem
Thanks for writing this up! I really appreciate effort-posts trying to show the efficacy of proposals, all the more so when the conclusion is negative. Thanks for pushing back against the file-drawer effect.