All of EliezerYudkowsky's Comments + Replies
Okay; I guess I was confused by your question because I thought I'd said that in the main doc.
To repeat and with added explanation: Only opinions from before ChatGPT count.
This is because ChatGPT moved the Overton window and changed which sorts of opinions would earn you the horror of contemptuous looks and lowered status, and my negative model of OpenPhil is that they miraculously arrived at a set of opinions which would balance which sort of looks they got from a weighted set of people they cared about. So whatever happened after the ChatGPT ...
My view of the tragedy of OpenPhil is indeed that they were very earnest people trying to figure out what was legit, but ended up believing stuff like "biologically anchored estimates of AI timelines" that were facially absurd and wrong and ultimately self-serving, because the problem "end up with beliefs about AI timelines that aren't influenced by what plays well with our funders and friends" was hard and frankly out of their league and OpenPhil did not know that it was a hard problem or treat it with what I would consider seriousness.
If you'd like to vi...
I'm not sure what's a truer analogy than static cling for hydrophobia as a force holding things together which the general audience has any experience with. Macroscopic experience of hydrophobia is, like, oil collecting on the surface of water, which isn't experienced as a binding force the way that static cling is.
I'm sort of skeptical that you could write something that works as science communication for a general audience, though lord knows I'm not necessarily succeeding either. The key valid ideas to be communicated are:
- There exists a level above biology for molecular systems, greatly superior in terms of strength and energy density. This sets a lower bound on how a very smart and uncaring entity could kill you, which looks like it attacking you with micron-diameter robots, which looks like everyone on Earth falling over dead in the same second.
- The de
To be clear, my main objection is that you have made statements that are implicitly or explicitly false. I go over each one in detail in the comment here. Yes, simplification is inevitable, but at many points you crossed the line into saying things that are flat out untrue.
I am confused by the pushback and downvotes in response to pointing this out. Do you not want to be making the strongest argument you can here?
I'm sort of skeptical that you could write something that works as science communication for a general audience
I don't think it's particula...
Why is flesh weaker than diamond? Diamond is made of carbon-carbon bonds. Proteins also have some carbon-carbon bonds! So why should a diamond blade be able to cut skin?
I reply: Because the strength of the material is determined by its weakest link, not its strongest link. A structure of steel beams held together at the vertices by Scotch tape (and lacking other clever arrangements of mechanical advantage) has the strength of Scotch tape rather than the strength of steel.
Or: Even when the load-bearing forces holding larg...
Okay. I'm going to take you at your word that you understand that biology is, at it's core, almost entirely built out of covalent bonds. In which case, I am utterly flabbergasted at the way you chose to communicate here.
I think the folded spaghetti gives the wrong impression (spaghetti is not hard to break apart). Let's instead talk about a structure which has at it's core a large steel wire (representing covalent bonds), where parallel sections are glued to each with extremely strong glue(that is obviously weaker than steel bonds) to build a backbon...
In this case, the positions from the last bullseyes become reversed. The doomer will argue that that AI might start off incapable, but will quickly evolve into a capable super-AI, following path A. Whereas I will retort that it might get more powerful, but that doesn’t guarantee it will ever actually end up being world domination worthy.
No, the doomer says, "If that AI doesn't destroy the world, people will build a more capable one." Current AIs haven't destroyed the world. So people are trying to build more capable ones.
There is some wei...
My point is that the trajectories affect the endpoints. You have fundamentally misunderstood my entire argument.
Say a rogue, flawed, AI has recently killed ten million people before being stopped. That results in large amounts of regulation, research, and security changes.
This can have two effects:
Firstly,(if AI research isn't shut down entirely), it makes it more likely that the AI safety problem will be solved due to increased funding and urgency.
Secondly, it makes the difficulty level of future takeover attempts greater, due to awarene...
The first object-level issue the author talks about is whether the brain is close to the Landaeur limit. No particular issue is cited, only that somebody else claimed a lot of authority and claimed I was wrong about something, what exactly is not shown.
The brain obviously cannot be operating near the Landaeur limit. Thousands of neurotransmitter molecules and thousands of ions need to be pumped back to their original places after each synaptic flash. Each of these is a thermodynamically irreversible operation and it staggers the imaginati...
Hi Eliezer. I actually do quite appreciate the reply because I think that if one writes a piece explaining why someone else is systematically in error, it's important that the other person can reply. That said . . .
You are misunderstanding the point about causal closure. If there was some isomorphic physical law, that resulted in the same physical states of affairs as is resulted in by consciousness, the physical would be causally closed. I didn't say that your description of what a zombie is was the misrepresentation. The poi...
I claim the word "disingenuous" is in fact breaking Forum norms, but not egregiously. It is unnecessarily unkind. I also do not think it rises to the level of formal warning. However, I deeply regret that Ted felt like taking a step back from the Forum on the basis of this comment. I think this shows the impact of unkind comments.
Somehow I never thought about it that way. Point conceded.
The analogy survives and if anything becomes more meaningful, but is now harder to explain to a general audience: After training humans exclusively on inclusive genetic fitness, with a correlation in the outer environment to high-calorie foods, humans ended up preferring something that didn't exist in the ancestral environment, lacks correlations to micronutrients that were reliably in ample supply in the ancestral environment and didn't need to be optimized over, has some resemblance to...
I think you have put your finger on a key aspect with the coldness requirement.
When ice cream is melted or coke is lukewarm, they both taste far too sweet. I've long had a hypothesis that we evolved some kind of rejection of foods that taste too sweet (at least in large quantity) and that by cooling them down, they taste less sweet (overcoming that rejection mechanism) but we still get increased reward when the sugar content enters our bloodstream. I feel that carbonation is similar (flat coke tastes too sweet), so that the cold and carbonation could...
FYI: IIRC/IIUC, Bryk is the one who made up the thing about my having a harem of submissive mathematicians whom I called my "math pets". This is false; people sufficiently connected within the community will know that it is false, not least since it'd be widely known and I wouldn't have denied it if it were true. I am not sure what to do about it simply, if someone's own epistemic location is such that my statements there are unknowable to them as being true.
It is known to me that Bryk has gone on repeating the "math pets" allegation, in...
IIRC, Jax is Bryk is the one who made up the "math pets" allegation against me, which hopefully everyone knows to be false. I don't know anything about the state of the rest of the allegations against Michael, but if I'm recalling correctly that Jax is that particular known-false-accuser, we probably want to subtract anything from Jax and then evaluate the rest of the list.
The usual argument, which I think is entirely valid, and has been delivered by famouser and more famously reputable people if you don't want to trust me about it, was named the "Gell-Mann Amnesia Effect" by Richard Feynman. Find something that you are really, truly an expert on. Find an article in TIME Magazine about it. Really take note of everything they get wrong. Try finding somebody who isn't an expert and see what their takeaways from the article were - what picture of reality they derive without your own expertise to guide th...
To be clear, at the risk of repetition, the question is not whether journalists should be considered reliable in their explanations of academic research. Although some journalists explain academic research quite well, others lack the training to do this as well as professional researchers. I would much rather turn to a colleague's blog than to a TIME magazine article to understand issues in academic philosophy, and I suspect that academics in most other fields would say the same.
The question is whether a major international publication should be cons...
I'd absolutely bring the same kind of skepticism. I would refuse to read a TIME expose of supposed abuses within LDS, because I would expect it to take way too much work to figure out what kind of remote reality would lie behind the epstemic abuses that I'd expect TIME (or the New York Times or whoever) would devise. If I thought I needed to know about it, I would poke around online until I found an essay written by somebody who sounded careful and evenhanded and didn't use language like journalists use, and there would then be a possibility th...
Would you trust a report from (say) the LDS church about the prevalence or non-prevalence of abuse in its ranks? [Continuing with example, not trying to say anything about the LDS church here.]
Organizations and movements certainly have an incentive to spin, minimize, and distort in their favor. And it's arguably easier to distort on the defensive side.
The epistemic challenge is that, unless abuse allegations result in judicial proceedings or some other public airing of evidence, we cannot realistically evaluate the underlying evidence directly. So if...
I've had worse experiences with coverage from professional journalists than I have from random bloggers. My standard reply to a journalist who contacts me by email is "If you have something you actually want to know or understand, I will answer off-the-record; I am not providing any on-the-record quotes after past bad experiences." Few ever follow up with actual questions.
A sincere-seeming online person with a blog can, any time they choose to, quote you accurately and in context, talk about the nuance, and just generally be truthful. Professional journalists exist in a much stranger context that would require much longer than this comment to describe.
If you want to insinuate that a major international publication is likely to be less reliable than a blog on issues of sexual harassment and abuse within the movement, it would be appropriate for you to write up the much longer description that you mention. This is a striking claim that is at odds with established views of what constitutes a trustworthy source. Most educated readers think that a publication such as TIME is among the most reliable sources that can be found on such a subject.
I don't deny that journalists sometimes have trouble understanding ...
I mean the human tendency, not the EA tendency. TIME does it because it's effective on their usual audience. EAs, evidently, have not risen above that.
If you think there's an actual problem, I think the correct avenue is doing a real investigation and a real writeup. Trying to "steelman" a media version of it, that is going to be incredibly and deliberately warped, adversarially targeted at exploiting the audience's underestimate of its warping by experienced adversaries, strikes me as a very wrong move. And it's just legit hard to ...
EAs, evidently, have not risen above that.
Again, I would love for you to provide an example of something unreasonable the community has done in response to the TIME article. As far as I can tell, the community is trying to figure out what is going on, but people's responses have generally been compassionate, open-minded, and reasonable.
If you think there's an actual problem, I think the correct avenue is doing a real investigation and a real writeup.
This would not be a good use of my time, in part because others are much better positioned than I am to do t...
I also attest that Aella is, if anything, severely underconveying the extent to which this central thesis is true. It's really really hard to convey until you've lived that experience yourself. I also don't know how to convey this to people who haven't lived through it. My experience was also of having been warned about it, but not having integrated the warnings or really actually understood how bad the misrepresentation actually was in practice, until I lived through it.
An attempt to help more EAs "get it": Almost every old-school vegan or vegetarian should instantly "get" that people will just lie about you. I think that most of us have experienced growing resentment toward us, built on false claims that we are rude about animal products, insulting, hate humans, etc. Or if we haven't experienced this ourselves, surely we have seen fellow veg*ns share stories like "I tried to be so polite at Thanksgiving.. I brought my own meal and didn't request anyone else make modifications, but now my family is saying I made a big fus...
You don't actually have to be famous in order to experience this; it's sufficient to be the kind of person who is easy to tell lies about. For instance, when I was in high school, other kids spread some really wild rumours about me, including that I had gotten in a fistfight with my English teacher and got away with it (without even getting detention), or that I cheated on all my tests. I did judo outside of school, and other kids apparently found that implausible enough that the majority of my school peers believed I was making it up and couldn't possibly...
Trying to "steelman" the work of an experienced adversary who relies on, and is exploiting, your tendency to undercompensate and not realize how distorted these things actually are - which is the practical, hard-earned knowledge that Aella is trying to propagate - seems like a mistake.
(Actually, trying to "steelman" is a mistake in general and you should focus on passing Ideological Turing Tests instead, but that's a much longer conversation.)
Let's be clear: the article in question was written by an internationally-renowned journalist in a major international publication. It went through all of the usual processes of careful research, writing, editing and fact-checking at the highest international standard. This article was written as a way to convey the experiences of women within the effective altruist community and to drive positive change in the way that women are treated within the movement.
This article is not the work of an "adversary" intent on taking someone down. They are not "exploiti...
I admit that my desire to steelman this article stems from my personal experiences in EA, and my general sense—as a woman in EA, who is friends with other women in EA, and has heard stories from still other women in EA—that this article does get at something important about the Bay Area EA community, even if it makes some important mistakes, many of which Aella helpfully identifies.
To steelman your comment, I assume by "your" you mean the EA community's (not my) "tendency to undercompensate and not realize how distorted these things actually are" (although...
Yet the tweets you linked were from 2/16 and 2/17.
Rational expectations doesn't mean "the alarmists are always right," and EMH doesn't imply that no one can profit helping correct the market.
The tweets you linked demonstrate the confusion at the time. Robin thought that China would be overwhelmed with COVID in a few months, while the rest of the world would be closing contact. In fact the rest of the world got overwhelmed with COVID and crashed their economies in just one month, while China contained it and kept its economy rolling for another two years. Rational expectations would've incorporated views like Robin's, but not parroted them. A plateau from early January and crash on 2/20 isn't inconsistent with that.
https://www.marketwatch.com/investing/index/djia
The Dow plateaued in early January and crashed starting Feb 20th, tracking rational expectations and three weeks ahead of media/mass awareness, which only caught up around March 12th
Trying again:
OP seems to ambiguate between two ideas, one true idea, and one false idea.
The true idea is that if Omega tells you personally that the world will end in 2030 with probability 1, you personally should not bother saving for retirement. Call this the Personal Idea.
The false idea is that if you believe in foomdoom, you should go long real interest rates and expect a market profit. Call this the Market Idea.
Intuitively, at least if you're swayed by this essay, the idea in Market probably seems pretty close to the idea in Personal. &nbs...
What would LTPZ or its post-facto equivalent have been doing around the time of the Cuban Missile Crisis? My model says 'no prediction'; they'll have done whatever. Afterwards somebody will make up a story about it in hindsight, but it is not the sort of thing where history says that long complicated analyses are remotely reliably good at doing it in advance.
In appendix 3 the authors cite a paper which looks at more-or-less this precise thing:
...Figure 4.6: Observed Treasury and aggregate-equity movements around President Kennedy’s 22 October 1962
Suppose you are one of the 0.1% of macro bonds traders familiar with Yudkowskian foom. You reason as follows: "Suppose that in the next 2 years, we get even more alarming news out of GPT-4 and successors. Suppose it's so incredibly alarming that 10% of macro traders notice, and then 10% of those hear about Yudkowskian foom scenarios. Putting myself into the shoes of one of those normie macro traders, I think I reason... that most actual normal people won't change their saving behavior any time soon, even if theoretically they ...
Trying again:
OP seems to ambiguate between two ideas, one true idea, and one false idea.
The true idea is that if Omega tells you personally that the world will end in 2030 with probability 1, you personally should not bother saving for retirement. Call this the Personal Idea.
The false idea is that if you believe in foomdoom, you should go long real interest rates and expect a market profit. Call this the Market Idea.
Intuitively, at least if you're swayed by this essay, the idea in Market probably seems pretty close to the idea in Personal. &nbs...
Beyond just taking vacation days, if you're a bond trader who believes in a very high chance of xrisk in the next five years it probably might make sense to quit your job and fund your consumption out of your retirement savings. At which point you aren't a bond trader anymore and your beliefs no longer have much impact on bond prices.
I'm a bit confused by this comment. Is the claim that you can't profit from taking out a loan that you have to pay back in 2040? The fact that you get money now and have to pay it back at a time when hypothetically money no longer matters seems like profit to me. If Omega whispers the truth into my ear that in 2040 there will be foom, then that's guaranteed profit.
People disagreeing, would you say why?
My guess for the pushback: 1 week before the end of the world, you think a sizable part of the population will notice and change their economic behavior drastically. I imagine this scenario contains a slow "attack" by AI that everyone sees coming?
(agree vote = yeah that is the pushback)
If you haven't extensively, successfully dealt with the media, someplace where the media do not start out nicely inclined towards you (i.e., your past media experience at the Center for Rare Diseases in Cute Puppies does not count), you are not qualified to give this advice. It should be given by somebody who understands how bad journalism gets and what needs to be done to avoid the usual and average negative outcome, or not at all.
It seems like setting ourselves up for selection bias if we take listen only to people with experience with "how bad journalism gets". We also want to get advice from people with good experiences with journalism, because they might be doing things that make them more likely to get good experiences, and presumably know about how to continue to go about having good experiences, having gotten them.
There may be some parts of EA where the media don't start out nicely inclined to the area at hand, but I think on many topics we might care to engage with the...
I think the sort of people who look at this advice and find that it sounded plausible to them, might want to first follow the rule of only taking advice that originated in actual lawyers, because they couldn't tell which nonlawyers had done real legal research. IDK, I don't know what it's like from the inside to read the original post and not scream.
Important notice to readers. Please vote up even though it is not very carefully argued here, because it may be important to some readers to read it immediately.
DO NOT FOLLOW THIS POST'S ADVICE. IT IS PROBABLY VERY BAD ADVICE FROM A LEGAL STANDPOINT. IF IT DOESN'T GET YOU IN TROUBLE IT WILL ONLY BE BECAUSE PEOPLE IGNORED YOUR LETTERS.
NEVER FOLLOW ADVICE LIKE THIS FROM PEOPLE WHO ARE NOT LAWYERS.
ONLY DO ANYTHING REMOTELY LIKE THIS IF YOU READ A POST FROM OPEN PHILANTHROPY'S LEGAL COUNSEL TELLING YOU TO DO IT.
My post from 4 months ago linked to a story about the lawyer, which is why I said I merely hinted at this point. The post from 2 months ago didn't expressly mention it, but a followup post definitely did in detail (I deleted the post soon thereafter because I got a few downvotes and I got nervous that maybe it was over the line).
Yeah, I think it's a severe problem that if you are good at decision theory you can in fact validly grab big old chunks of deontology directly out of consequentialism including lots of the cautionary parts, or to put it perhaps a bit more sharply, a coherent superintelligence with a nice utility function does not in fact need deontology; and if you tell that to a certain kind of person they will in fact decide that they'd be cooler if they were superintelligences so they must be really skillful at deriving deontology from decision theory and therefore they...
This strikes me as a bad play of "if there was even a chance". Is there any cognitive procedure on Earth that passes the standard of "Nobody ever might have been using this cognitive procedure at the time they made $mistake?" That more than three human beings have ever used? I think when we're casting this kind of shade we ought to be pretty darned sure, preferably in the form of prior documentation that we think was honest, about what thought process was going on at the time.
Why require surety, when we can reason statistically? There've been maybe ten comparably-sized frauds ever, so on expectation, hardline act utilitarians like Sam have been responsible for 5% of the worst frauds, while they represent maybe 1/50M of the world's population (based on what I know of his views 5-10yrs ago). So we get a risk ratio of about a million to 1, more than enough to worry about.
Anyway, perhaps it's not worth arguing, since it might become clearer over time what his philosophical commitments were.
Maybe they weren't familiar with the overwhelming volume of previous historical incidents, hadn't had their brains process history or the news as real events rather than mythology, or were genuinely unsure about how often these sorts of things happened in real life rather than becoming available on the news. I'm guessing #2.
The point there isn't so much, "He could not have had any EA thoughts in his head at all", which I doubt is really true - though also there could've just been pressure from coworkers, and office politics around it, resolving in something like the Future Fund so that they were doing anything. My point is just that this nightmare is probably not one of a True Sincere Committed EA Act Utilitarian doing these things; that person would've tried to take more money off the table, earlier, for the Future Fund. Needing an e-sports site named after your ...
I was passing through the Bahamas and asked if FTX wanted me to talk to the EAs they had on fellowships there. They paid for my hotel room and an Airbnb when the hotel got full, for a week. I'm not sure but I don't think I remember getting to see SBF at all while I was at the hotel. Didn't go swimming or sunning or any such because I am not a very outdoors person. It does not seem entirely accurate to characterize this as "was hosted by SBF in the Bahamas".
The Future Fund basically turned down all my ideas until the regrantor progra...
I agree that if I, personally, had steered SBF into crypto, and uncharacteristically failed to add on a lot of "hey but please don't scam people, only do this if you find a kind of crypto you can feel good about" I might consider myself more at fault. I even think that the Singer side of EA in fact does less talking about deontology, less writing of fiction that exemplifies the feelings and reasoning behind that deontology, less cautioning of people against twisting up their brains by chasing good ideas; on my view, the Singer side explicitly starts ...
I agree that if I, personally, had steered SBF into crypto, and uncharacteristically failed to add on a lot of "hey but please don't scam people, only do this if you find a kind of crypto you can feel good about" I might consider myself more at fault.
Given how big of a role EA apparently had in the origin of Alameda (Singh says in the Sequoia puff piece that it wouldn’t have started without EA), there very likely are many members of the community who offered more encouragement and/or didn’t give as many warnings as they should have.
I don’t know what poi...
Okay; I agree then that it's reasonable to say of Ben Delo that Hayes and cofounders were accused of trying to defraud two early investors, that Ben Delo is accused of taunting them with a meme, and that they settled out of court.
I do note that this is pretty different from what Vaughan was previously accusing Delo of, which sounded pretty plausibly like a "victimless crime".
Yes, it's in the complaint, here's a screenshot (where the CEO also talks about it being easier to bribe officials in the Seychelles with "a coconut": https://i.imgur.com/qd6X4LC.png
I don't know anything about the plaintiffs, but I assume BitMEX's lawyers certainly thought a jury would find them credible, and that's why they decided to settle for $44M.
The BitMEX indictment is here: https://www.justice.gov/usao-sdny/pr/founders-and-executives-shore-cryptocurrency-derivatives-exchange-charged-violation#_ftn1
The government's indictment shouldn't be treated as the "truth," of course, but the facts are damning and extremely shady - these guys were claiming they were operating a foreign entity for non-US citizens (to trade crummy crypto derivatives), but they were actually doing it from an office in Manhattan and selling to people in the US and helping them conceal it. They also made sure that BitMEX skirted ...
I got a one-time gift of appreciated crypto, not through MIRI, part of whose purpose as I understood it was to give me enough of a savings backstop (having in previous years been not paid very much at all) that I would feel freer to speak my mind or change my mind should the need arise.
I have of course already changed MIRI's public mission sharply on two occasions, the first being when I realized in 2001 that alignment might need to be a thing, and said so to the primary financial supporter who'd previously supported MIRI (then SIAI) on the premise of char... (read more)