I'm asking how it would change your personal assessment, so for questions about things like whether ML is a science or not you could apply your own view on those issues. Perhaps we could say something like NeurIPS to make it concrete. I would guess most of reviewers have an ML background and no or limited social science experience. Would that change your view?
It’s not based on what would typically be considered acceptable scientific research. METR’s paper would not likely pass peer review at a credible scientific journal, if it were submitted.
If METR were to publish their timelines research in a peer reviewed ML conference, would that change your view?
I’d like to see more of the rigor-seeking anti-longtermists taking the side of the neartermist EAs against their speculative socialist critics.
I think your factional infighting angle isn't really substantiated. Many rigor-seeking anti-longtermists seem to believe that risks from AI are overblown (I disagree with this) and their rigor-related critiques are connected to this. I don't see why they have any obligation to side against EA's socialist critics, for two reasons:
They could believe that there is rigorous evidence for socialism or some version of the socialist critique of EA.
They could believe that AI risk has more traction within EA compared to socialists critiques of EA (a correct assessment) and therefore prioritize critiquing AI risk advocates rather than socialists.
Personally, I'd like to see EAs, rationalists, and critics of both of these movements arguing the actual fucking point more, instead of making meta-arguments about why the people they disagree with are violating some newly invented standard. AI risk advocates don't need to convince rigor-seeking anti-longtermists that actually AI risk is the most scientifically rigorous conclusion to have ever existed in order to decide that they are concerned about AI risk. Nor do rigor-seeking anti-longtermists need to stamp out all socialist critics of EA before critiquing EA for a lack of rigor. We can all just argue the actual fucking point instead.
Reading this gave me occaion to read Austin's previous comments on the subject that are linked in the article. Not sure if I read them at the time but regardless, reading or re-reading them makes me feel more willing to go harder on Austin than I need in the comments on the other post.
Below is directed @Austin:
But from EA folks, this behavior strikes me as cowardly, coldhearted, opportunistic, bandwagon-y, two-faced and distasteful. I am extremely confused, because these EA leaders are some of the smartest and “good-est” people in the world, whose work I respect and admire, who have shaped the way I think. So it’s very possible that I’m just in the wrong here, but…
What type of galaxy-brained, tying-yourself-in-knots, motivated-reasoning-infused bullshit reasoning does it take to accuse business associates or work collaborators of SBF of disloyalty and two-facedness for not defending him, while hiring a person who knowingly engaged in a criminal conspiracy and then testified against their co-conspirator only after the crime came to light, in exchange for a significantly reduced sentence, the literal platonic ideal of an act of disloyalty?
I don't believe that your primary interest in this whole situation is loyalty, at least in the sense of the type of interpersonal loyalty that I consider a virtue, vs loyalty to an in-group. I also don't believe that your primary reason for hiring Ellison is a belief that she is among the best for the position, as you claim in your other post. You may actually believe she is, but I don't believe, based on statements like the one I quote here, that this the main factor driving your decision. I think the main factor is the viewpoint which you discuss in the post linked above, and your announcement post gives emphasizes the "she's the best for the job" reason because you know that is more defensible. To me, this seems like you claiming to oppose PR-y type actions by EAs while engaging in them yourself. If you really believe the ideas you claim in the post defending SBF, as you seem to, say it with your fucking chest!
Would you be open to a public debate on this topic, potentially for stakes? I'm open to wagering on a debate judged by lesswrong or EA forum vote totals of an exchange at least 10 messages long, details TBD. I'd prefer a topic specific to AI safety but open to one that is general as well.
t's plausible to me that some frontier lab employees working on AI capabilities, may be grappling with situations that rhymes with Caroline's at FTX
I agree with this and thought about a similar connection when I read your OP, but I feel like you are badly mis-analyzing the situation. I think my other comment (which I know you replied to, thanks!) is super relevant here.
In that comment, I articulate the idea that your worldview strongly emphasizes incentivzing and valuing ambition and risk-taking, and I contrast this with a view that incorporates "red lines". The relevance of this to the AI situation is two-fold:
Many AI employees who may be considering leaving AI labs probably didn't violate any common-sense morality red lines. Notably red lines that derive from "common-sense morality" are not necesarily 1-1 with expected harm, which I think is a fact that EAs/rationalist don't seem to have entirely internalized in the discussions around AI.
A huge fucking reason why people joined the labs in the first place and continue to join or work at them is the exact ambition-based mentality you seem so enamored with. Just take a look at some of the Greg Brockman documents that came out of the litigation with Musk, or observe that its Sillicon Valley, not some other culture, where these labs are coming out of. Bro, the call is coming from inside the house!!!!
The sense I get from reading this is that Ellison's history with FTX was a significant positive reason in your mind for hiring her, whereas people might have generally expected this to be a negative factor. This reminded me of your talk with Elizabeth Van Nostrand on the topic of FTX. There, you expressed similar conflicted feeling around the events of FTX. One of the things you expressed was feelings of concern around the community reaction in regard to a sense of loyalty, that the reaction created an impression that people might not stand by someone if things went badly. I feel like there is some tension with the idea that people from the EA community are engaging in backstabbing in terms of their reaction to the FTX fiasco, while presumably you don't view testifying against SBF as similarly implicating the concern about loyalty.
To me, this reminds me of a attitude that I associate with Silicon Vally VC culture and see coming through strongly in the sentiments around this issue. It seems to me like the feeling/pirnciple you are invoke here is more like downside protection than loyalty. I think there is a feeling prevalent in SV that entrenched interests are often the opponents of change and innovation, and that process and rules primarily serve those entrenched interests. Thus, brave innovators need to be willing to challenge and sometimes even break rules in order to change the world for the better. I think this is the idea you get at with your weak doge/buff shiba comment. A follow on from this that seems to have caught on among people of this subculture is that people who engage in these brave risk-taking effors deserve praise and high status, and ought to be protected and admired even if they fail.
You can try to cast that as being about protecting people from the mob (and this is explicitly brought up in the conversation), but to me that doesn't really seem to be the core value being served. Yes innovators and founders may get unfairly attacked by a public that doesn't really understand the underlying issues or a bureaucracy that is controlled by legacy players, but from what I can tell the core "ask" of this SV subculture is to support founders mostly regardless of whether evidence that those things are occuring is present. We are kind of meant to assume that of course that must be happening if someone is critizing a founder/ambitious innovator. The core value here is ambition, not loyalty.
In the talk, You and Elizabeth explore the tensions between your "winning" frame and her "truthseeking" one. I think the analysis I give above fits squarely within that, but I would offer a something of a synthesis that I think you two miss, and that is generally underappreciated in the wider rationality/EA ecosystem:
I believe your "winning" frame is well-modeled by the description I give above, but what of the "truthseeking" frame? Later in the talk you two consider the question of whether "truthseeking" often cause people/orgs to need to slow down. I think this does identify a core problem with "truthseeking" as it is often conceptualized. If I'm trying to make a new cure for cancer, and I'm 95% confident it is safe and effective, am I duty-bound for truthseeking reasons to never let anyone use it until I'm 99.999999% certain? "Truthseeking" sounds great in theory when you are just reading posts about meta-honesty or whatever, but when it comes to application it seems to really struggle with managing tradeoffs. You end up with a situation where people who are doing there best to navigate tough choices can too easily get called out for being "insufficiently truthseeking" for not meeting the arbitrary standards of someone else.
I think there is a reasonable synthesis of these that is actually fairly commonly held outside of EA/rationality, which is that you should try to win, according to your own definition of winning, while treating certain ethical contraints like truthfulness/honesty (in contrast to "truthseeking") as red lines that you need to stay on the correct side of. This solves the problem with truthseeking in that ambitious people have a safe harbor if they stay on the right side of those lines, will still reigning in bad behavior by unscrupulous innovators.
Game theory gets brought up in the talk as well. Within my interpretation of your framework, it makes sense to hire Ellison on these grounds. She was in a sense a victim of the powers that be because she was an ambitious person and the system failed to provide downside protection. You can provide that by giving her a job. Under this view, it is very important to do this in order to send the signal that innovation and ambitition are valued and people will still support you if you fail.
The trouble with this within the "truthfulness" framework that I articulate here is that it fails to disincentivize crossing red lines, and arguably even incentivizes doing so. There have been several incidences of fraud or related conduct in SV VC-funded startups in recent history, suggesting that this may a phenomenon that is genuinely playing out in practice.
It isn't sufficient in my view merely that a positive EV action exists, you also need to be able to identity at least one such action, or else you are in the situation where longtermism is only theoretical and doesn't actually recommend certain actions. This makes the difficulty of modeling the future more of an issue that your post suggests because in realistic situations you need to estimate these EVs, you don't just know them.
The relevant uncertainty is in the estimated EV of a specific actions or intervention, it isn't sufficient for your argument merely to know that positive EV actions exists, or even that one is in some class of interventions (like those that plausibly reduce extinction risk), for longtermism to suggest a specific action is good. It requires considering the EV of that specific action, estimation error included. The difficulty of performing that estimation is therefore a difficulty for longtermism.
While the future is very hard to predict, it would be very surprising if there was no possible action which benefitted the long-term future in expectation. And if there are actions like that, then they swamp the expected value of other actions. It is an error to infer from the fact that the future is very unpredictable that we’re totally in the dark about which actions can make it better.
I think this needs additional assumptions to make it go through in realistic situations. There is a bit of an errors-in-variables problem. I think as stated, the argument only works if we assume actual knowledge of expected values, but in realistic situations these would need to be estimated. For reasons similar to what is discussed here and here, I think you can have a situation where there are actions that have positive EV on the future, but your estimation + intervention selection process can't reliably identify them, and integrating out this estimation/selection based uncertainty can change the overall dynamic. Basically, your expected outcome isn't the population expected value of a given action, the expected value you realize needs to take into account estimation/selection.
This doesn't mean that there aren't positive EV estimation/selection procedures, just that the existence of positive EV actions isn't sufficient for the arugment to go through.
On the bad faith/"truth seeking" point, I've also noted some issues in the way "truth seeking" is used in a previous post, and thinking about this case gave me an idea. It seems like there is a general phenomenon in EA/rationalist discourse where intent gets obscured or ignored somehow. Perhaps not surprising for intellectual communities that are very into consequentialism?
I think the effect of using the "truth seeking" terminology is to confuse multiple possibilities around intent:
Lying: intentional
Insufficient rigor/evidence/etc.: can be an unintentional mistake
Callousness about the truth: I think people often feel like even if someone isn't lying, they can demonstrate a disregard for truth that feels like its intentionally misleading
Being "insufficiently truth seeking" could refer to any of these, and thus using the phrase fails my principle of clarity. It also becomes strongly subject to motivate reasoning or motte/bailey dynamics because the meaning can shift among these different meanings.
I feel like something similar has happened here with the whole "hard to dispell misunderstandings" thing. Tskeen laterally, it doesn't make a whole lot of sense to soft ban someone for this. Is the implicict message that only easy to dispell misunderstandings are allowed? Surely not. But it makes more sense if you imagine its implicitly standing in for a spectrum of actions based on intention:
Bad faith: intentionally obscuring your real views, often with the goal of making them harder to respond to.
Genuine mistake: unintentional, even of hard to dispell
Game playing: being coy or cagey about what your views really are or in some way deliberately making your position confusing or hard to respond to.
The last one I think is kind of what Said was being accused of in the post referenced above? But IMO its extremely clear that you haven't been doing anything intentionally misleading. I think this is an instance of concerns about "epistemics" being used in an unproductive way that creates a lack of clarity around intent.
I'm asking how it would change your personal assessment, so for questions about things like whether ML is a science or not you could apply your own view on those issues. Perhaps we could say something like NeurIPS to make it concrete. I would guess most of reviewers have an ML background and no or limited social science experience. Would that change your view?
If METR were to publish their timelines research in a peer reviewed ML conference, would that change your view?
I think your factional infighting angle isn't really substantiated. Many rigor-seeking anti-longtermists seem to believe that risks from AI are overblown (I disagree with this) and their rigor-related critiques are connected to this. I don't see why they have any obligation to side against EA's socialist critics, for two reasons:
They could believe that there is rigorous evidence for socialism or some version of the socialist critique of EA.
They could believe that AI risk has more traction within EA compared to socialists critiques of EA (a correct assessment) and therefore prioritize critiquing AI risk advocates rather than socialists.
Personally, I'd like to see EAs, rationalists, and critics of both of these movements arguing the actual fucking point more, instead of making meta-arguments about why the people they disagree with are violating some newly invented standard. AI risk advocates don't need to convince rigor-seeking anti-longtermists that actually AI risk is the most scientifically rigorous conclusion to have ever existed in order to decide that they are concerned about AI risk. Nor do rigor-seeking anti-longtermists need to stamp out all socialist critics of EA before critiquing EA for a lack of rigor. We can all just argue the actual fucking point instead.
Reading this gave me occaion to read Austin's previous comments on the subject that are linked in the article. Not sure if I read them at the time but regardless, reading or re-reading them makes me feel more willing to go harder on Austin than I need in the comments on the other post.
Below is directed @Austin:
What type of galaxy-brained, tying-yourself-in-knots, motivated-reasoning-infused bullshit reasoning does it take to accuse business associates or work collaborators of SBF of disloyalty and two-facedness for not defending him, while hiring a person who knowingly engaged in a criminal conspiracy and then testified against their co-conspirator only after the crime came to light, in exchange for a significantly reduced sentence, the literal platonic ideal of an act of disloyalty?
I don't believe that your primary interest in this whole situation is loyalty, at least in the sense of the type of interpersonal loyalty that I consider a virtue, vs loyalty to an in-group. I also don't believe that your primary reason for hiring Ellison is a belief that she is among the best for the position, as you claim in your other post. You may actually believe she is, but I don't believe, based on statements like the one I quote here, that this the main factor driving your decision. I think the main factor is the viewpoint which you discuss in the post linked above, and your announcement post gives emphasizes the "she's the best for the job" reason because you know that is more defensible. To me, this seems like you claiming to oppose PR-y type actions by EAs while engaging in them yourself. If you really believe the ideas you claim in the post defending SBF, as you seem to, say it with your fucking chest!
Would you be open to a public debate on this topic, potentially for stakes? I'm open to wagering on a debate judged by lesswrong or EA forum vote totals of an exchange at least 10 messages long, details TBD. I'd prefer a topic specific to AI safety but open to one that is general as well.
I agree with this and thought about a similar connection when I read your OP, but I feel like you are badly mis-analyzing the situation. I think my other comment (which I know you replied to, thanks!) is super relevant here.
In that comment, I articulate the idea that your worldview strongly emphasizes incentivzing and valuing ambition and risk-taking, and I contrast this with a view that incorporates "red lines". The relevance of this to the AI situation is two-fold:
Many AI employees who may be considering leaving AI labs probably didn't violate any common-sense morality red lines. Notably red lines that derive from "common-sense morality" are not necesarily 1-1 with expected harm, which I think is a fact that EAs/rationalist don't seem to have entirely internalized in the discussions around AI.
A huge fucking reason why people joined the labs in the first place and continue to join or work at them is the exact ambition-based mentality you seem so enamored with. Just take a look at some of the Greg Brockman documents that came out of the litigation with Musk, or observe that its Sillicon Valley, not some other culture, where these labs are coming out of. Bro, the call is coming from inside the house!!!!
Note: "You" = Austin in this comment
The sense I get from reading this is that Ellison's history with FTX was a significant positive reason in your mind for hiring her, whereas people might have generally expected this to be a negative factor. This reminded me of your talk with Elizabeth Van Nostrand on the topic of FTX. There, you expressed similar conflicted feeling around the events of FTX. One of the things you expressed was feelings of concern around the community reaction in regard to a sense of loyalty, that the reaction created an impression that people might not stand by someone if things went badly. I feel like there is some tension with the idea that people from the EA community are engaging in backstabbing in terms of their reaction to the FTX fiasco, while presumably you don't view testifying against SBF as similarly implicating the concern about loyalty.
To me, this reminds me of a attitude that I associate with Silicon Vally VC culture and see coming through strongly in the sentiments around this issue. It seems to me like the feeling/pirnciple you are invoke here is more like downside protection than loyalty. I think there is a feeling prevalent in SV that entrenched interests are often the opponents of change and innovation, and that process and rules primarily serve those entrenched interests. Thus, brave innovators need to be willing to challenge and sometimes even break rules in order to change the world for the better. I think this is the idea you get at with your weak doge/buff shiba comment. A follow on from this that seems to have caught on among people of this subculture is that people who engage in these brave risk-taking effors deserve praise and high status, and ought to be protected and admired even if they fail.
You can try to cast that as being about protecting people from the mob (and this is explicitly brought up in the conversation), but to me that doesn't really seem to be the core value being served. Yes innovators and founders may get unfairly attacked by a public that doesn't really understand the underlying issues or a bureaucracy that is controlled by legacy players, but from what I can tell the core "ask" of this SV subculture is to support founders mostly regardless of whether evidence that those things are occuring is present. We are kind of meant to assume that of course that must be happening if someone is critizing a founder/ambitious innovator. The core value here is ambition, not loyalty.
In the talk, You and Elizabeth explore the tensions between your "winning" frame and her "truthseeking" one. I think the analysis I give above fits squarely within that, but I would offer a something of a synthesis that I think you two miss, and that is generally underappreciated in the wider rationality/EA ecosystem:
I believe your "winning" frame is well-modeled by the description I give above, but what of the "truthseeking" frame? Later in the talk you two consider the question of whether "truthseeking" often cause people/orgs to need to slow down. I think this does identify a core problem with "truthseeking" as it is often conceptualized. If I'm trying to make a new cure for cancer, and I'm 95% confident it is safe and effective, am I duty-bound for truthseeking reasons to never let anyone use it until I'm 99.999999% certain? "Truthseeking" sounds great in theory when you are just reading posts about meta-honesty or whatever, but when it comes to application it seems to really struggle with managing tradeoffs. You end up with a situation where people who are doing there best to navigate tough choices can too easily get called out for being "insufficiently truthseeking" for not meeting the arbitrary standards of someone else.
I think there is a reasonable synthesis of these that is actually fairly commonly held outside of EA/rationality, which is that you should try to win, according to your own definition of winning, while treating certain ethical contraints like truthfulness/honesty (in contrast to "truthseeking") as red lines that you need to stay on the correct side of. This solves the problem with truthseeking in that ambitious people have a safe harbor if they stay on the right side of those lines, will still reigning in bad behavior by unscrupulous innovators.
Game theory gets brought up in the talk as well. Within my interpretation of your framework, it makes sense to hire Ellison on these grounds. She was in a sense a victim of the powers that be because she was an ambitious person and the system failed to provide downside protection. You can provide that by giving her a job. Under this view, it is very important to do this in order to send the signal that innovation and ambitition are valued and people will still support you if you fail.
The trouble with this within the "truthfulness" framework that I articulate here is that it fails to disincentivize crossing red lines, and arguably even incentivizes doing so. There have been several incidences of fraud or related conduct in SV VC-funded startups in recent history, suggesting that this may a phenomenon that is genuinely playing out in practice.
It isn't sufficient in my view merely that a positive EV action exists, you also need to be able to identity at least one such action, or else you are in the situation where longtermism is only theoretical and doesn't actually recommend certain actions. This makes the difficulty of modeling the future more of an issue that your post suggests because in realistic situations you need to estimate these EVs, you don't just know them.
The relevant uncertainty is in the estimated EV of a specific actions or intervention, it isn't sufficient for your argument merely to know that positive EV actions exists, or even that one is in some class of interventions (like those that plausibly reduce extinction risk), for longtermism to suggest a specific action is good. It requires considering the EV of that specific action, estimation error included. The difficulty of performing that estimation is therefore a difficulty for longtermism.
I think this needs additional assumptions to make it go through in realistic situations. There is a bit of an errors-in-variables problem. I think as stated, the argument only works if we assume actual knowledge of expected values, but in realistic situations these would need to be estimated. For reasons similar to what is discussed here and here, I think you can have a situation where there are actions that have positive EV on the future, but your estimation + intervention selection process can't reliably identify them, and integrating out this estimation/selection based uncertainty can change the overall dynamic. Basically, your expected outcome isn't the population expected value of a given action, the expected value you realize needs to take into account estimation/selection.
This doesn't mean that there aren't positive EV estimation/selection procedures, just that the existence of positive EV actions isn't sufficient for the arugment to go through.
Thanks for your kind words.
On the bad faith/"truth seeking" point, I've also noted some issues in the way "truth seeking" is used in a previous post, and thinking about this case gave me an idea. It seems like there is a general phenomenon in EA/rationalist discourse where intent gets obscured or ignored somehow. Perhaps not surprising for intellectual communities that are very into consequentialism?
I think the effect of using the "truth seeking" terminology is to confuse multiple possibilities around intent:
Lying: intentional
Insufficient rigor/evidence/etc.: can be an unintentional mistake
Callousness about the truth: I think people often feel like even if someone isn't lying, they can demonstrate a disregard for truth that feels like its intentionally misleading
Being "insufficiently truth seeking" could refer to any of these, and thus using the phrase fails my principle of clarity. It also becomes strongly subject to motivate reasoning or motte/bailey dynamics because the meaning can shift among these different meanings.
I feel like something similar has happened here with the whole "hard to dispell misunderstandings" thing. Tskeen laterally, it doesn't make a whole lot of sense to soft ban someone for this. Is the implicict message that only easy to dispell misunderstandings are allowed? Surely not. But it makes more sense if you imagine its implicitly standing in for a spectrum of actions based on intention:
Bad faith: intentionally obscuring your real views, often with the goal of making them harder to respond to.
Genuine mistake: unintentional, even of hard to dispell
Game playing: being coy or cagey about what your views really are or in some way deliberately making your position confusing or hard to respond to.
The last one I think is kind of what Said was being accused of in the post referenced above? But IMO its extremely clear that you haven't been doing anything intentionally misleading. I think this is an instance of concerns about "epistemics" being used in an unproductive way that creates a lack of clarity around intent.