All of Sharmake's Comments + Replies

I basically grant 2, sort of agree with 1, and drastically disagree with three (that timelines will be long.)

Which makes me a bit weird, since while I do have real confidence in the basic story that governments are likely to influence AI a lot, I do have my doubts that governments will try to regulate AI seriously, especially if timelines are short enough.

Yeah, at least several comments have much more severe issues than tone or stylistic choices, like rewording ~every claim by Ben, Chloe and Alice, and then assuming that the transformed claims had the same truth value as the original claim.

I'm in a position very similar to Yarrow here: While I think Kat Woods has mostly convinced me that the most incendiary claims are likely false, and I'm sympathetic to the case for suing Ben and Habryka, there was dangerous red flags in the responses, so much so that I'd stop funding Nonlinear entirely, and I think it's quite bad that Kat Woods responded the way they did.

I unendorsed primarily because apparently, the board didn't fire because of safety concerns, though I'm not sure this is accurate.

It seems like the board did not fire Sam Altman for safety reasons, but instead for other reasons instead. Utterly confusing, and IMO demolishes my previous theory, though a lot of other theories also lost out.

Sources below, with their archive versions included:

https://twitter.com/norabelrose/status/1726635769958478244

https://twitter.com/eshear/status/1726526112019382275

https://archive.is/dXRgA

https://archive.is/FhHUv

While I generally agree that they almost certainly have more information on what happened, which is why I'm not really certain on this theory, my main reason here is that for the most part, AI safety as a cause basically managed to get away with incredibly weak standards of evidence for a long time, until the deep learning era in 2019-, especially with all the evolution analogies, and even now it still tends to have very low standards (though I do believe it's slowly improving right now). This probably influenced a lot of EA safetyists like Ilya, who almos... (read more)

If Ilya can say "we're pushing capabilities down a path that is imminently highly dangerous, potentially existentially, and Sam couldn't be trusted to manage this safely" with proof that might work - but then why not say that?

I suspect this is due to the fact that quite frankly, the concerns they had about Sam Altman being unsafe on AI basically had no evidence except speculation from the EA/LW forum, which is not enough evidence at all in the corporate world/legal world, and to be quite frank, the EA/LW standard of evidence on AI risk being a big deal ... (read more)

My general thoughts on this can be stated as: I'm mostly of the opinion that EA will survive this, bar something massively wrong like the board members willfully lying or massive fraud from EAs, primarily because most of the criticism is directed to the AI safety wing, and EA is more than AI safety, after all.

Nevertheless, I do think that this could be true for the AI safety wing, and they may have just hit a key limit to their power. In particular, depending on how this goes, I could foresee a reduction in AI safety power and influence, and IMO this was completely avoidable.

Yeah, this is one of the few times where I believe that the EAs on the board likely overreached here, because they probably didn't give enough evidence to justify their excoriating statement there that Sam Altman was dishonest, and he might be coming back to lead the company.

I'm not sure how to react to all of this, though.

Edit: My reaction is just WTF happened, and why did they completely play themselves? Though honestly, I just believe that they were inexperienced.

The Bay Area rationalist scene is a hive of techno-optimisitic libertarians.[1] These people have a negative view of state/government effectiveness at a philosophical and ideological level, so their default perspective is that the government doesn't know what it's doing and won't do anything. [edit: Re-reading this paragraph it comes off as perhaps mean as well as harsh, which I apologise for]

Yeah, I kinda of have to agree with this, I think the Bay Area rationalist scene underrates government competence, though even I was surprised at how little politi... (read more)

Okay, my crux is that the simplicity/Kolmogorov/Solomonoff prior is probably not very malign, assuming we could run it, and in general I find the prior not to be malign except for specific situations.

This is basically because it relies on the IMO dubious assumption that the halting oracle can only be used once, and notably once we use the halting/Solomonoff oracle more than once, the Solomonoff oracle loses it's malign properties.

More generally, if the Solomonoff Oracle is duplicatable, as modern AIs generally are, then there's a known solution to mitigate... (read more)

I want to flag that I see quite a lot of inappropriate binarization happening, and I generally see quite a lot of dismissals of valid third options.

Either they take the extinction risks seriously, or they don't.

There are other important possibilities, like a potential belief in AI progress helping or solving the existential risk, thinking that the intervention of increasing AI progress is actually the best strategy, etc. More generally, once we make weaker or no assumptions about AI risk, we no longer obtain the binary you've suggested.

So this doesn't ... (read more)

Yeah, I'd really like to know how they'd respond to information that says that they'd have to stop doing something that would go against their incentives, like accelerating AI progress.

I don't think it's very likely, but given the incentives at play, it really matters that the organization will actually be able to at least seriously consider the possibility that the solution to AI safety might be something that they aren't incentivized to do, or have anti-incentives to doing.

The basic reasoning is that SGD is an extremely powerful optimizer, and even the imperfections of SGD in real life that mesa-optimizers can use are detectable without much interpretability progress at all. Also, there is an incentive by capabilities groups to improve SGD, so we have good reason to expect that these flaws become less worrisome over time.

In particular, it is basically immune to acausal trade setups or blackmail setups by mesa-optimizers.

Some choice quotes from Beren's post below:

The key intuition is that gradient descent optimizes the enti

Even though I don't think EA needs to totally replicate outside norms, I do agree that there are good reasons why quite a few norms exist.

I'd say the biggest norms from outside that EA needs to adopt are less porous boundaries on work/dating, and importantly actually having normalish pay structures/work environments.

Are you saying AIs trained this way won’t be agents?

Not especially. If I had to state it simply, it's that massive space for instrumental goals isn't useful today, and plausibly in the future for capabilities, so we have at least some reason to not worry about misalignment AI risk as much as we do today.

In particular, it means that we shouldn't assume instrumental goals to appear by default, and to avoid overrelying on non-empirical approaches like your intuition or imagination. We have to take things on a case-by-case basis, rather than using broad jud... (read more)

I agree, but that implies pretty different things than what is currently being done, and still implies that the danger from AI is overestimated, which bleeds into other things.

Basically, kind of. The basic issue is that instrumental convergence, and especially effectively unbounded instrumental convergence is a central assumption of why AI is uniquely dangerous, compared to other technologies like biotechnology. And in particular, without the instrumental convergence assumption, or at least with the instrumental convergence assumption being too weak to make the case for doom, unlike what Superintelligence told you, matters a lot because it kind of ruins a lot of our inferences of why AI would likely doom us, like deception or un... (read more)

I mean, regardless of how much better their papers are in the meantime, does it seem likely to you that those labs will solve alignment in time if they are racing to build bigger and bigger models?

Basically, yes. This isn't to say that we aren't doomed at all, but contrary to popular beliefs of EAs/rationalists, the situation you gave actually has a very good chance, like >50% of working, for a few short reasons:

1. Vast space for instrumental convergence/instrumental goals aren't incentivized in current AI, and in particular, the essentially unbounded

I want to be clear for the record here that this is enormously wrong, and Greg Colbourn's advice should not be heeded unless someone else checks the facts/epistemics of his announcements, due to past issues with his calls for alarm.

I definitely agree with this, and I'm not very happy with the way Omega solely focuses on criticism, at the very least without any balanced assessment.

And given the nature of the problem, some poor initial results should be expected, by default.

I actually would find this at least somewhat concerning, because selection bias/selection effects are my biggest worry with smart people working in an area. If a study area is selected based upon any non-truthseeking motivations, or if people are pressured to go along with a view for non-truthseeking reasons, then it's very easy to land into nonsense, where the consensus is based totally on selection effects, making them useless to us.

There's a link to the comment by lukeprog below on the worst case scenario for smart people being dominated by selection ef... (read more)

This. I generally also agree with your 3 observations, and the reason I was focusing on truth seeking is because my epistemic environment tends to reward worrying AI claims more than it probably should due to negativity bias, as well as looking at AI Twitter hype.

What kind of a breakthrough are you envisaging? How do we get from here to 100% watertight alignment of an arbitrarily capable AGI?

Scalable alignment is the biggest way to align a smarter intelligence.

Now, Pretraining from Human Feedback showed that at least for one of the subproblems of alignment, outer alignment, we managed to make the AI more aligned as it gets more data.

If this generalizes, it's huge news, as it implies we can at least align an AI's goals with human goals as we get more data. This matters because it means that scalable alignment isn... (read more)

No, at the higher end of probability. Things still need to be done. It does mean we should stop freaking out every time a new AI capability is released, and importantly it means we probably don't need to go to extreme actions, at least right away.

The reason I'm so confident in this is that right now is because I assess a significant probability of the AI developments starting with GPT-4 to be a hype cycle, and in particular I am probably >50% confident in the idea that most of the flashiest stuff on AI will prove to be over hyped.

In particular, I am skeptical of the general hype on AI right now, and that a lot of capabilities tests essentially test it on paper tests, not real world tasks, which are much less Goodhartable than paper tests

Now I'd agree with you conditioning on the end game being 2... (read more)

I think that we are pretty close to the end game already (maybe 2 years), and that there's very little chance for alignment to be solved / x-risk reduced to acceptable levels in time.

I believe with 95-99.9% probability that this is purely hype, and that we will not in fact see AI that radically transforms the world or that is essentially able to do every task that automates physical infrastructure in 2 years.

Given this, I'd probably disagree with this:

Have you considered that the best strategy now is a global moratorium on AGI (hard as that may be)? I

The point is this distorts the apparent balance of reason - maybe this is like Marxism, or NGDP targetting, or Georgism, or general semantics, perhaps many of which we will recognise were off on the wrong track.

I do note one is not like the others here. Marxism is probably way more wrong than any of the other beliefs, and I feel like the inclusion of the others rather weakens the case here.

Honestly, a lot of the problems from politics stems from both it's totalizing nature, comparable to strong longtermism, plus emotion hampers more often than it helps in political discussions compared to longtermism.

I'd say that if EA can't handle politics in the general forum, then I think a subforum for EA politics should be made. Discussions about the politics of EA or how to effectively do politics can go there.

Meanwhile, the general EA forum can simply ban political posts and discussions.

Yes, it's a strong measure to ban politics here. But bluntly, in ... (read more)

Finally, I do think that there is a risk of updating too quickly the other way. Back on the original post, some users have responded this comment saying that it's 'entirely correct'[1], but I don't think it's reasonable to view Expo's piece a 'major misrepresentation' of what happened - their reporting on the case seems to have been accurate. While 'Option 1' seems to be what happened there is still a question of how the grant made it to stage 5 out of 7 on FLI's grant-making procedure. It's not the major scandal we feared, but it's ramifications are more

My big concern is that permanent harm could be suffered by either EA or it's championed causes. Somewhat like how transhumanism became tarred with the brush of racism and eugenics, I worry things like AI safety or X-risk work could be viewed in the same light as racism. And there may be much more at stake than people realize.

The problem is that even without a hinge of history, our impacts, especially in a longtermism framework, are far far larger than previous generations, and we could very well lose all that value if EA or it's causes become viewed as badly as say eugenics or racism was.

PR/political

you want people to believe a certain thing (even if it's something you yourself sincerely believe), in this case that EA is not racist it's about managing impressions and reputations (e.g. EA's reputation as not racist) Your initial comment (and also the Bostrom email statement) both struck me as "performative" in how they demonstrated really harsh and absolute condemnation ("absolutely horrifying", "[no] place in this community", "recklessly flawed and reprehensible" – granted that you said "if true", but the tone and other comments seemed

Psychologists know IQ is a somewhat mysterious measure (no, scoring lower on an IQ test does not necessary mean a person is "more stupid"). It is affected by things like income shifts across generations and social position. For Bostrom to even have that opinion as an educated 23-year-old was bad, but to not unequivocably condemn it today - despite the harm it can clearly cause - seems even worse.

I disagree, because I think the evidence from psychology is that IQ is a real measure of intelligence, and while a lot of old tests had high cultural biases, ... (read more)

Only a small red flag, IMO, because it's rather easy to convince people of alluring falsehoods, and not so easy to convince people of uncomfortable truths.

I don't think there's been a huge scandal involving Will? Sure, there are questions we'd like to see him openly address about what he could have done differently re FTX - and I personally am concerned about his aforementioned influence because I don't want anyone to have that much - but very few if any people here seem to believe he's done anything in seriously bad faith.

I was imagining a counterfactual world where William Macaskill did something hugely wrong.

And yeah come to think of it, selection may be quite a bit stronger than I think.

Putting Habryka's claim another way: If Eliezer right now was involved in a huge scandal like say SBF or Will Macaskill was, then I think modern LW would mostly handle it pretty fine. Not perfectly, but I wouldn't expect nearly the amount of drama that EA's getting. (Early LW from the 2000s or early 2010s would probably do worse, IMO.) My suspicion is that LW has way less personal drama over Eliezer than say, EA would over SBF or Nick Bostrom.

Those with high quality epistemics usually agree on similar things

On factual questions, this is how it should be, and this matters. Putting it another way, it's not a problem for EAs to come to agree on factual questions, without more assumptions.

Yeah, given that no violence by the people criticizing Bostrom's apology is happen, unlike the actual struggle sessions, I don't understand how they're very comparable.

Also: while those with high quality epistemics usually agree on similar things is a distortion making the argument personal, about people, in reality yes, good reasoning often converges to similar conclusions

This. Aumann's Agreement Theorem tells us that Bayesian that have common priors and trust each other to be honest cannot disagree.

The in practice version of this is that a group agreeing on similar views around certain subjects isn't automatically irrational, unless we have outside evidence or one of the conditions is wrong.

Also, as far as "social capital," comments from this forum are regularly reposted as evidence of what "EA thinks" of a given controversy. If an apology is insufficient and we are all silent, the inference that we think the apology sufficient will be drawn.

And arguably rightly, IMO.

If I were to extract generalizable lessons from the FTX, the major changes I would make are:

1. EA should stay out of crypto, until and unless the situation improves to the extent that it doesn't have to rely on speculators. One big failure is EAs thought they could invest in winner stocks more than other investors.

2. Good Governance matters. By and large, EA failed at basic governance tasks, and I think governance needs to be improved. My thoughts are similar to this post:

https://forum.effectivealtruism.org/posts/sEpWkCvvJfoEbhnsd/the-ftx-crisis-highlig... (read more)

My views on what EA should learn from this event is the following:

1. EA needs to articulate what moral views or moral values it will not accept in the pursuit of it's goals. I don't believe EA can consider every moral point valid due to the Paradox of Tolerance. Thus, moderators and administrators need to start working on what values or moral viewpoints it will not accept, and it will need to be willing to ban or cancel people who violate this policy.

2. EA has an apology problem. Titotal has a good post on it, but a lot of apologies tend to be bad. Linking

I don't see it that way. Lots of relatively normal folks put money into FTX. Journalists and VCs were very positive about SBF/FTX.

This. I do think that blaming rationalist culture is mostly a distraction, primarily because way too much normie stuff promoted SBF.

I had a very different opinion of the whole crypto train (that is, crypto needs to at least stop having real money, if not flat out banned altogether.)

Yes, EA failed. But let's be more careful about suggesting that normies didn't fail here.

I agree that EA failed pretty hard here. My big disagreements are probably on why EA failed, not that EA failed to prevent harm.

Many people outside the rat sphere in my life think the whole FTX debacle, for instance, is ridiculous because they don't find SBF convincing at all. SBF managed to convince so many people in the movement of his importance because of his ability to expound and rationalize his opinions on many different topics very quickly. This type of communication doesn't get you very far with normal, run of the mill folks.

I ignored SBF and the crypto crowd, however I disagree with this primarily because I think this is predictably overrating how much you wouldn't fal... (read more)

I don't exactly agree with the case that cultural knowledge is really important like Henrich wants to say, though I do credit cultural knowledge for increasing returns to scale.

Basically the problem with neglectedness is that it assumes strictly declining returns like a logarithmic scale. But if the problem has the quality "the more the merrier", or increasing returns to scale, then neglectedness is a problem. In other words, leverage matters.

I assume you consider Blank Slate doctrine false. Do you believe communism would've worked out in a world where it was true? (My view is that most or all of the problems with communism would remain.)

Yeah, this. The real issues with communism ultimately come down to ignoring thermodynamics exists. Once you accept that idea, a lot of other false ideas from communism starts to make more sense.

I also think that, even for a high decoupler (which I consider myself to be, though as far as I know I'm not on the autism spectrum) the really big taboos - like race and intelligence - are usually obvious, as is the fact that you're supposed to be careful when talking about them. The text of Bostrom's email demonstrates he knows exactly what taboos he's violating.

And honestly, I think this is a great taboo for many reasons. I'd argue this is one of the more intelligent taboos here by the left.