I've been trying to go through this sequence premise by premise and failing to come up with a satisfactory rebuttal. The reason is that roughly speaking, I find it to be a correct account about a wide range of potential claims to knowledge about far-future outcomes.
Nonetheless, I expect that certain outcomes are predictable nonetheless, contingent on antecedent events. These are roughly the ones the EA community has focused on (with a few minor exceptions, e.g. the value and sign of a very near-term AI Pause). The badness of extinction. The persistence of wild animal suffering, in a range of futures. The potential for many sentient digital minds. The expected persistence and scale of factory farming. Etc...(further specificity would require a full post).
So why these ones and not others? I think the sequence has generally been rather irrationally pessimistic in regards to certain claims to knowledge, which it has obscured by staying away from object-level claims. For instance:
"Consider your favorite intervention, let’s say, advocating for digital mind welfare. And imagine you become aware of some new pessimistic hypothesis. E.g., advocating for digital mind welfare could make it easier for misaligned AIs to take over by gaining humans’ sympathy (see Fenwick). Should you update toward more pessimism about the other possibilities in the catch-all? How much more?"
This just seems like a very implausible scenario! Yes, we may be unaware of the more plausible complementary issues that make concern valid, but our inability to come up with these issues (given the effort, importantly!) should be taken as evidence against them. I realize saying "seems like a very implausible scenario" is not a great account of a credible intuitive process, but there's clearly way more mechanistic intricacy in the misaligned ASI scenario. The misalignment, the usefulness for the ASI, the weird edge-case were this was actually an instrumental factor, etc. I'm not sure how helpful it is to go through this in greater detail; I suspect it is more plausible that this type of back-lash effect is just there to serve a psychological function, the mere conception being the satisfying evidence and not the strength of the reason. This is a type of argumentation that should be itself suspect.
It's not surprising that the sequence is pessimistic in these ways: I think that it's a fairly well-evidenced sociological phenomenon now that people respond pretty doomishly to far-future scenarios. Dystopias over utopias, and disempowering cluelessness (and contrived back-lash effects) over individual agency. But we should keep in mind that some form of eutopia[1] is by considerable margins the most probable. And further, that most interventions will tend to either have small effects or do nothing. I think these are good hedges against going off-kilter in one way or another.
I agree with some of this. But let me attempt a conciliatory take: less of forecasting money and effort should go to platforms and tournaments, but more should go to identifying existing, nascent forecasts (people using the word "probably" or "unlikely" about empirical matters) and creating markets (even unsubsidized Manifold markets would be helpful on the margin). I think it would be very helpful for someone to go through popular EA forum posts and org research documents and do this systematically.
"Super dead" is a bit of an exaggeration, but there is less activity than there should be. The main issue is pretty apparent to me though: there isn't sufficient cross-posting to the forum. There's tons of good stuff on substack, but also on various org websites and blogs.
I don't think other causes are nearly as important as this.
I may be missing important context, but I think you are mistaken here on the norms at hand in this case. I do applaud you for helping your friend out; that makes you a good friend. But opportunities for people to be altruistic are completely unbounded; I could find hundreds of similar asks for help in a 5 minute google search, most of which aren't distinctively "good opportunities". If this wasn't a personal request, but instead calling for donations to a related cause you were making a case for, that would be fine. I think highlighting personal requests for help is permissible and is even virtuous interpersonal behavior between friends and family. People reach out on facebook pages like this all the time. But it just looks like spam or emotional manipulation when posted on online forums dedicated to other purposes, with colleagues or strangers. Hopefully this helps! This can definitely be a confusing discourse norm contextually.
For context: Clara is right, there is good experimental evidence that this occurs in online comment forums. This is on top of the simple mechanism that more highly upvoted content is more likely to be seen for various reasons.
I'd assume this holds true for EA forum content. I do the same thing @Toby Tremlett🔹 is describing to some extent, but I'd be surprised if my system 2 thinking outweighs my system 1 on net in this regard. I suspect I personally do this most with very low Karma posts, which I neglect to upvote because of a vague embarrassment over the possibility of promoting content with some flaw I missed.
Due to Value Lock-in, TAI poses a time constraint for farmed animal social progress.
I do not expect most issues to be resolved before this time, due to technological limitations, heightened barriers to social change relative to historic movements, and increasing developing world meat consumption.
If we open this up to wild animals rather than just farmed, net-negative outcomes are much more assured.
I do tend to favor longer AGI/TAI timelines than many for roughly these reasons. But I don't think you are exactly right about the AI data access trend. For one, whether or not me or Americans at large are "happy to give an ASI full autonomous power to gather such biomedical data", China will be.
I tentatively I expect capabilities with real-world economic importance to come to some extent in the US as well, even if the most radical and transformative stuff requires further integration into the physical world for modeling. And at that point there may simply be a iterative process of greater and greater integration, as public perception improves and dependence increases. The complication here is moral backlash of some sort, which I note you've written about before. I agree that this is plausible, I simply wouldn't call it probable. Things look more bi-modal to me; most likely we get the outcome I've described above (mild harms could still be disregarded by China), or we get a longer slow down before curing aging.
Semantic quibble: I think most people, myself included, simply define ASI as either encompassing those capabilities or being sufficient at recursive self-improvement such it will possess those capabilities in short order.
If your point is primarily that the existing AI paradigm is inadequate, I would tend to agree. There's also a distinct question of what an intelligence explosion looks like; it may well be that tedious real-world experimentation is necessary for these sorts of biomedical advances, which takes time. That too is a compelling possibility; but I would expect it in a decade at most and certainly quicker than human R&D can advance.
It might genuinely be the time to boycott Chat GPT and start campaigns targeting corporate partners. But this isn't yet obvious. Even if so, what would be the appropriate concrete and reasonable asks? I think there is a bit of epistemic crisis emerging at the moment. If there's a case to be made, it needs to be made sooner rather than latter. And then we need coordination.
I've been trying to go through this sequence premise by premise and failing to come up with a satisfactory rebuttal. The reason is that roughly speaking, I find it to be a correct account about a wide range of potential claims to knowledge about far-future outcomes.
Nonetheless, I expect that certain outcomes are predictable nonetheless, contingent on antecedent events. These are roughly the ones the EA community has focused on (with a few minor exceptions, e.g. the value and sign of a very near-term AI Pause). The badness of extinction. The persistence of wild animal suffering, in a range of futures. The potential for many sentient digital minds. The expected persistence and scale of factory farming. Etc...(further specificity would require a full post).
So why these ones and not others? I think the sequence has generally been rather irrationally pessimistic in regards to certain claims to knowledge, which it has obscured by staying away from object-level claims. For instance:
"Consider your favorite intervention, let’s say, advocating for digital mind welfare. And imagine you become aware of some new pessimistic hypothesis. E.g., advocating for digital mind welfare could make it easier for misaligned AIs to take over by gaining humans’ sympathy (see Fenwick). Should you update toward more pessimism about the other possibilities in the catch-all? How much more?"
This just seems like a very implausible scenario! Yes, we may be unaware of the more plausible complementary issues that make concern valid, but our inability to come up with these issues (given the effort, importantly!) should be taken as evidence against them. I realize saying "seems like a very implausible scenario" is not a great account of a credible intuitive process, but there's clearly way more mechanistic intricacy in the misaligned ASI scenario. The misalignment, the usefulness for the ASI, the weird edge-case were this was actually an instrumental factor, etc. I'm not sure how helpful it is to go through this in greater detail; I suspect it is more plausible that this type of back-lash effect is just there to serve a psychological function, the mere conception being the satisfying evidence and not the strength of the reason. This is a type of argumentation that should be itself suspect.
It's not surprising that the sequence is pessimistic in these ways: I think that it's a fairly well-evidenced sociological phenomenon now that people respond pretty doomishly to far-future scenarios. Dystopias over utopias, and disempowering cluelessness (and contrived back-lash effects) over individual agency. But we should keep in mind that some form of eutopia[1] is by considerable margins the most probable. And further, that most interventions will tend to either have small effects or do nothing. I think these are good hedges against going off-kilter in one way or another.
a term I take to include pretty low-value-but-positive futures, e.g. average utilitarian outcomes
I agree with some of this. But let me attempt a conciliatory take: less of forecasting money and effort should go to platforms and tournaments, but more should go to identifying existing, nascent forecasts (people using the word "probably" or "unlikely" about empirical matters) and creating markets (even unsubsidized Manifold markets would be helpful on the margin). I think it would be very helpful for someone to go through popular EA forum posts and org research documents and do this systematically.
"Super dead" is a bit of an exaggeration, but there is less activity than there should be. The main issue is pretty apparent to me though: there isn't sufficient cross-posting to the forum. There's tons of good stuff on substack, but also on various org websites and blogs.
I don't think other causes are nearly as important as this.
I may be missing important context, but I think you are mistaken here on the norms at hand in this case. I do applaud you for helping your friend out; that makes you a good friend. But opportunities for people to be altruistic are completely unbounded; I could find hundreds of similar asks for help in a 5 minute google search, most of which aren't distinctively "good opportunities". If this wasn't a personal request, but instead calling for donations to a related cause you were making a case for, that would be fine. I think highlighting personal requests for help is permissible and is even virtuous interpersonal behavior between friends and family. People reach out on facebook pages like this all the time. But it just looks like spam or emotional manipulation when posted on online forums dedicated to other purposes, with colleagues or strangers.
Hopefully this helps! This can definitely be a confusing discourse norm contextually.
For context: Clara is right, there is good experimental evidence that this occurs in online comment forums. This is on top of the simple mechanism that more highly upvoted content is more likely to be seen for various reasons.
I'd assume this holds true for EA forum content. I do the same thing @Toby Tremlett🔹 is describing to some extent, but I'd be surprised if my system 2 thinking outweighs my system 1 on net in this regard. I suspect I personally do this most with very low Karma posts, which I neglect to upvote because of a vague embarrassment over the possibility of promoting content with some flaw I missed.
Is he going to starve if I stop reading posts?! I'm too scared to leave the forum now.
Due to Value Lock-in, TAI poses a time constraint for farmed animal social progress.
I do not expect most issues to be resolved before this time, due to technological limitations, heightened barriers to social change relative to historic movements, and increasing developing world meat consumption.
If we open this up to wild animals rather than just farmed, net-negative outcomes are much more assured.
I do tend to favor longer AGI/TAI timelines than many for roughly these reasons. But I don't think you are exactly right about the AI data access trend. For one, whether or not me or Americans at large are "happy to give an ASI full autonomous power to gather such biomedical data", China will be.
I tentatively I expect capabilities with real-world economic importance to come to some extent in the US as well, even if the most radical and transformative stuff requires further integration into the physical world for modeling. And at that point there may simply be a iterative process of greater and greater integration, as public perception improves and dependence increases. The complication here is moral backlash of some sort, which I note you've written about before. I agree that this is plausible, I simply wouldn't call it probable. Things look more bi-modal to me; most likely we get the outcome I've described above (mild harms could still be disregarded by China), or we get a longer slow down before curing aging.
Semantic quibble: I think most people, myself included, simply define ASI as either encompassing those capabilities or being sufficient at recursive self-improvement such it will possess those capabilities in short order.
If your point is primarily that the existing AI paradigm is inadequate, I would tend to agree. There's also a distinct question of what an intelligence explosion looks like; it may well be that tedious real-world experimentation is necessary for these sorts of biomedical advances, which takes time. That too is a compelling possibility; but I would expect it in a decade at most and certainly quicker than human R&D can advance.
It might genuinely be the time to boycott Chat GPT and start campaigns targeting corporate partners. But this isn't yet obvious. Even if so, what would be the appropriate concrete and reasonable asks? I think there is a bit of epistemic crisis emerging at the moment. If there's a case to be made, it needs to be made sooner rather than latter. And then we need coordination.