All of gwern's Comments + Replies

I don't think it's odd at all. As the Bloomberg article notes, this was in response to the martyrdom of Saint Altman, when everyone thought the evil Effective Altruists were stealing/smashing OA for no reason and destroying humanity's future (as well as the fortune of many of the individuals involved, to a degree few of them bothered to disclose) and/or turning it over to the Chinese commies. An internal memo decrying 'Effective Altruism' was far from the most extreme response at the time; but I doubt Gomez would write it today, if only because so much new... (read more)

Good report overall on tacit knowledge & biowarfare. This is relevant to the discussion over LLM risks: the Aum Shinrikyo chemist could make a lot of progress by reading papers and figuring out his problems as he went, but the bacteriologist couldn't figure out his issues for what seems like what had been a viable plan to weaponize & mass-produce anthrax but where lack of feedback led it to fail. Which does sound like something that a superhumanly-knowledgeable (but not necessarily that intelligent) LLM could help a lot with simply by pattern-matching and making lists of suggestions for things that are to the human 'unknown unknowns'.

If a crazy person wants to destroy the world with an AI-created bioweapon

Or, more concretely, nuclear weapons. Leaving aside regular full-scale nuclear war (which is censored from the graph for obvious reasons), this sort of graph will never show you something like Edward Teller's "backyard bomb", or a salted bomb. (Or any of the many other nuclear weapon concepts which never got developed, or were curtailed very early in deployment like neutron bombs, for historically-contingent reasons.)

There is, as far as I am aware, no serious scientific doubt that ... (read more)

That was covered in my 2023-11-25 comment.

Hm, maybe it was common knowledge in some areas? I just always took him for being concerned. There's not really any contradiction between being excited about your short-term work and worried about long-term risks. Fooling yourself about your current idea is an important skill for a researcher. (You ever hear the joke about Geoff Hinton? He suddenly solves how the brain works, at long last, and euphorically tells his daughter; she replies: "Oh Dad - not again!")

Ilya has always been a doomer AFAICT, he was just loyal to Altman personally, who recruited him to OA. (I can tell you that when I spent a few hours chatting with him in... 2017 or something? a very long time ago, anyway - I don't remember him dismissing the dangers or being pollyannaish.) 'Superalignment' didn't come out of nowhere or surprise anyone about Ilya being in charge. Elon was... not loyal to Altman but appeared content to largely leave oversight of OA to Altman until he had one of his characteristic mood changes, got frustrated and tried to tak... (read more)

1. I haven't seen any coverage of the double structure or Anthropic exit which suggests that Amodei helped think up or write the double structure. Certainly, the language they use around the Anthropic public benefit corporation indicates they all think, at least post-exit, that the OA double structure was a terrible idea (eg. see the end of this article).

2. You don't know that. They seem to have often had near majorities, rather than being a token 1 or 2 board members.

   By most standards, Karnofsky and Sutskever are 'doomers', and Zillis is likely a 'doomer'

EDIT: this is going a bit viral, and it seems like many of the readers have missed key parts of the reporting. I wrote this as a reply to Wei Dai and a high-level summary for people who were already familiar with the details; I didn't write this for people who were unfamiliar, and I'm not going to reference every single claim in it, as I have generally referenced them in my prior comments/tweets and explained the details & inferences there. If you are unaware of aspects like 'Altman was trying to get Toner fired' or pushing out Hoffman or how Slack was... (read more)

By the time Musk (and Altman et al) was starting OA, it was in response to Page buying Hassabis. So there is no real contradiction here between being spurred by Page's attitude and treating Hassabis as the specific enemy. It's not like Page was personally overseeing DeepMind (or Google Brain) research projects, and Page quasi-retired about a year after the DM purchase anyway (and about half a year before OA officially became a thing).

The discussion of the Abbey, er, I mean, 'castle', has been amusing for showing how much people are willing to sound off on topics from a single obviously-untrustworthy photograph. Have you ever seen a photograph of the interior or a layout? No, you merely see the single aerial real estate brochure shot using a telephoto zoom lenses framed as flatteringly as possible to include stuff that isn't even the Abbey - like that turreted 'castle' you see in the photo up above isn't even part of the Abbey - because that's an active church, All Saints Church!* (Real... (read more)

Current reporting is that 'EAs out of the board' (starting with expelling Toner for 'criticizing' OA) was the explicit description/goal told to Sutskever shortly before, with reasons like being to avoid 'being painted in the press as “a bunch of effective altruists,” as one of them put it'.

Whatever the nature of Q*, there is not much evidence that it could have prompted the Altman firing. It's not clear why some very early preliminary results about a Q* as described would prompt a firing nor why the firing would be so abrupt or when it was if the research happened months ago (and Altman was alluding to it publicly weeks ago), while Sutskever's involvement & the exact timing of the firing appear to be adequately explained by other issues.

As there is still nothing leaking or confirming Q*, I'm increasingly skeptical of its relevance - for ... (read more)

Maybe? I can't easily appreciate such a usecase because I always want to save any excerpts I find worth excerpting. Are there a lot of people who want that? If that's the idea, I guess the "About Highlights" dialogue needs a bit of documentation to explain the intended (and unintended) uses. At least, anyone who doesn't realize that the annotations are ephemeral, because they aren't enough of a web dev to understand 'What you save is stored only on your specific browser locally' is as much of a bug as it is a feature, is in for a bad time when their annotations inevitably get deleted...

First issue post.

I like it overall.

But I have a lot of questions about the 'highlight' feature: aside from the many teething problems Said has already documented, which bugs doubtless will be fixed, I don't understand what the usecase is, compared to other web annotation systems like Hypothesis - so it stores arbitrary ranges of text, saving them to, I assume, dangerously ephemeral browser LocalStorage where they will be unpredictably erased in a few hours / days / weeks, and unavailable on any other device presumably. Why do I want this? They aren't clipped to something ... (read more)

I agree. I've already started to develop a bit of an instinctive 'ugh' reaction to random illustrations, even ones without obvious generative model tell-tales or the DALL-E watermark.

It's comparable to how you feel when you notice that little '© Getty Images' or a Memphis style image, and realize your time & (mental) bandwidth was wasted by the image equivalent of an emoji. It's not that they look bad, necessarily, but they increasingly signify 'cheap' and 'tacky'. (After all, if this monkey pox image can be generated by a prompt of 12 redundant words,... (read more)

For those wondering why we needed a stylish magazine for provocative rationalist/EA nonfiction when Works In Progress is pretty good too, Scott Alexander says

Works In Progress is a Progress Studies magazine, I'm sure these two movements look exactly the same to everyone on the outside, but we're very invested in the differences between them.

What disease would you seek FDA approval for? "I sleep more than 4 hours a day" is not a recognized disease under the status quo. (There is the catch-all of 'hypersomnia', but things like sleep apnea or neurodegenerative disorders or damage to clock-keeping neurons would not plausibly be treated by some sort of knockout-mimicking drug.)

One downside you don't mention: having a Wikipedia article can be a liability when editors are malicious, for all the reasons it is a benefit when it is high-quality like its popularity and mutability. A zealous attacker or deletionist destroying your article for jollies is bad, but at least it merely undoes your contribution and you can mirror it; an article being hijacked (which is what a real attacker will do) can cause you much more damage than you would ever have gained as it creates a new reality which will echo everywhere.

My (unfortunately very long... (read more)

Note: most of the discussion of this is currently on LW.

The Wall Street Journal article How a Public School in Florida Built America’s Greatest Math Team (non-paywalled version) describes how a retired Wall Street bond trader built a math team that has won 13 of the last 14 national math championships at an otherwise unremarkable high school. His success is not based on having a large budget, but rather on thinking differently and building an ecosystem.

The otherwise unremarkable high school has pick of the litter from everyone living around one of the largest universities in the country which is <5 miles ... (read more)

The above seems voluminous and I believe this is the written output with the goal of defending a person.

Yes, much like the OP is voluminous and is the written output with the goal of criticizing a person. You're familiar with such writings, as you've written enough criticizing me. Your point?

Yeah, no, it's the exact opposite.

No, it's just as I said, and your Karnofsky retrospective strongly supports what I said. (I strongly encourage people to go and read it, not just to see what's before and after the part He screenshots, but because it is a good r... (read more)

Not sure why this is on EAF rather than LW or maybe AF, but anyway. I find this interesting to look at because I have been following Eliezer's work since approximately 2003 on SL4, and so I remember this firsthand, as it were. I disagree with several of the evaluations here (but of course agree with several of the others - I found the premise of Flare to be ludicrous at the time, and thankfully, AFAICT, pretty much zero effort went into that vaporware*):

• calling LOGI and related articles 'wrong' because that's not how DL looks right now is itself wrong.

this would lead to catastrophic forgetting

It's unclear that this is true: "Effect of scale on catastrophic forgetting in neural networks". (The response on Twitter from catastrophic forgetting researchers to the news that their field might be a fake field of research, as easily solved by scale as, say, text style transfer, and that continual learning may just be another blessing of scale, was along the lines of "but using large models is cheating!" That is the sort of response which makes me more, not less, confident in a new research direction. New AI ... (read more)

There are limits, however: scaling alone would not allow Gato to exceed expert performance on diverse tasks, since it is trained to imitate the experts rather than to explore new behaviors and perform in novel ways.

Imitation can exceed experts or demonstrations: note that Gato reaches >=100%† expert performance on something like a third of tasks (Figure 5), and does look like it exceeds the 2 robot experts in Figure 10 & some in Figure 17. This is a common mistake about imitation learning and prompt engineering or Decision Transformer/Trajectory... (read more)

Well, you know what the stereotype is about women in Silicon Valley high tech companies & their sock needs... (Incidentally, when I wrote a sock-themed essay, which was really not about socks, I was surprised how many strong opinions on sock brands people had, and how expensive socks could be.)

If you don't like the example 'buy socks', perhaps one can replace it with real-world examples like spending all one's free time knitting sweaters for penguins. (With the rise of Ravelry and other things, knitting is more popular than it has been in a long time.)... (read more)

It’s hard to imagine a newsletter that could have picked out that paper at the time as among the most important of the hundreds included. For comparison, I think probably that at the time, there was much more hype and discussion of Hinton and students’ capsule nets (also had a NIPS 2017 paper).

People at the time thought it was a big deal: https://twitter.com/Miles_Brundage/status/1356083229183201281 Even the ones who were not saying it would be "radically new" or "spicy" or "this is going to be a big deal" or a "paradigm shift" were still at least askin... (read more)

I dunno if it's that hard. Comparisons are an old and very well-developed area of statistics, if only for use in tournaments, and you can find a ton of papers and code for pairwise comparisons. I have some & a R utility in a similar spirit on my Resorter page. Compared (ahem) to many problems, it's pretty easy to get started with some Elo or Bradley-Terry-esque system and then work on nailing down your ordinal rankings into more cardinal stuff. This is something where the hard part is the UX/UI and tailoring to use-cases, and too much attention to the statistics may be wankery.

I humbly request a photo of Buck et al in a van with the caption "Get in loser we're going saving".

Absolutely. But you know you are relying on obscurity and relatively modest cost there, and you keep that in mind when you comment. Which is fine. Whereas if you thought that it was secure and breaking it came at a high cost (though it was in fact ~5 seconds of effort away), you might make comments you would not otherwise. Which is less fine.

I would strongly advise closing the commenting loophole then, if that was never intended to be possible. The only thing worse than not having security/anonymity is having the illusion of security/anonymity.

as he himself explains.

Yes, he does claim it. So, why did you do it? Why did you post his whole username, when I did not and no one could figure out who it was from simply 'Mark'?

I point out that is a reasonable characterization that all the effects/benefits of calling out Mark accrue to Gwern by the device of using Mark's first name, yet he can escape a charge of "doxxing", by the same.

Absolutely. I did not dox him, and I neither needed nor wanted to. I did what illustrated my point with minimum harm and I gained my desired benefits that way. This ... (read more)

Oh, the whole story is strictly speaking unnecessary :). There are disjunctively many stories for an escape or disaster, and I'm not trying to paint a picture of the most minimal or the most likely barebones scenario.

The point is to serve as a 'near mode' visualization of such a scenario to stretch your mind, as opposed to a very 'far mode' observation like "hey, an AI could make a plan to take over its reward channel". Which is true but comes with a distinct lack of flavor. So for that purpose, stuffing in more weird mechanics before a reward-hacking twis... (read more)

It might help to imagine a hard takeoff scenario using only known sorts of NN & scaling effects... (LW crosspost, with >82 comments)

It Looks Like You're Trying To Take Over The World

In A.D. 20XX. Work was beginning. "How are you gentlemen !!"... (Work. Work never changes; work is always hell.)

Specifically, a MoogleBook researcher has gotten a pull request from Reviewer #2 on his new paper in evolutionary search in auto-ML, for error bars on the auto-ML hyperparameter sensitivity like larger batch sizes, because more can be different and there's h

So the question about whether a self-supervised RL agent like a GPT-MuZero-hybrid of some sort could pollute its own dataset makes me think that because of self-supervision, even discussing it in public is a minor infohazard: because discussing the possibility of a treacherous turn increases the probability of a treacherous turn in any self-supervised model trained on such discussions, even if only a tiny part of its corpus.

GPT is trained to predict the next word. This is a simple-sounding objective which induces terrifyingly complex capabilities. To help ... (read more)

Yes, the brain is sparse and semi-modularized, but it'd be hard to really call it more 'brain-like' than dense models. Brains have all sorts of very long range connections in a small-world topology, where most of the connections may be local but there's still connections to distant parts, and those are important; distant brain regions can also communicate and be swapped in and out as the brain recurs and ponders. The current breed of MoEs along the lines of Switch Transformer don't do any of that. They do a single pass, and each module is completely local ... (read more)

(If anyone asks, say 'PASTA' was designed as an allusion to Strega Nona.)

1. Your example doesn't make sense to me. If Bob is not providing any money and cannot 'personally lose the cash' and is never 'any worse off' because he just resells it, what is he doing, exactly? Extending Anne some sort of disguised interest-free loan? (Guaranteed and risk-free how?) Why can't he be replaced by a smart contract if there are zero losses?

   It seems like in any sensible Paul-like capitalist system, he must be providing money somewhere in the process - if only by eating the loss when no one shows up to buy it at the same or higher price! If Bo

The impact certificate is resold when someone wants to become the owner of it and pays more than the current owner paid for it; that's just built-in, like a Harberger tax except there's no ongoing 'tax'. (I thought about what if you made a tax that paid to the creator - sort of like an annuity? "Research X is great, as a reward, here's the NPV of it but in the form of a perpetuity." But the pros and cons were unclear to me.) The current owner has no choice about it, and if they want to keep owning it, well, they can then just buy it back at their real valu... (read more)

Certificates seem like a nice match for NFTs because if you are serious about the status/prestige thing, you do want a global visible registry so you can brag about what impacts you retrocausally funded; and for creators, this makes a lot more sense than doing one-off negotiations over, like, email.* I was thinking about Harberger taxes on NFTs and how to ensure that NFT collectibles can always be transferred without needing a tax and ratcheting up price as a mechanisms, and that doesn't work because of wash trades with oneself (esp powered by flash loans)... (read more)

I mostly agree with that with the further caveat that I tend to think the low value reflects not that ML is useless but the inertia of a local optima where the gains from automation are low because so little else is automated and vice-versa ("automation as colonization wave"). This is part of why, I think, we see the broader macroeconomic trends like big tech productivity pulling away: many organizations are just too incompetent to meaningful restructure themselves or their activities to take full advantage. Software is surprisingly hard from a social and ... (read more)

Lousy paper, IMO. There is much more relevant and informative research on compute scaling than that.

I think your confusion with the genetics papers is because they are talking about _effective_ population size (N~e~), which is not at all close to 'total population size'. Effective population size is a highly technical genetic statistic which has little to do with total population size except under conditions which definitely do not obtain for humans. It's vastly smaller for humans (such as 10^4) because populations have expanded so much, there are various demographic bottlenecks, and reproductive patterns have changed a great deal. It&ap... (read more)

This seems like a retread of Bostrom's argument that, despite astronomical waste, x-risk reduction is important regardless of whether it comes at the cost of growth. Does any part of this actually rely on Roodman's superexponential growth? It seems like it would be true for almost any growth rates (as long as it doesn't take like literally billions or hundreds of billions of years to reach the steady state).

“Recent GWASs on other complex traits, such as height, body mass index, and schizophrenia, demonstrated that with greater sample sizes, the SNP h2 increases. [...] we suspect that with greater sample sizes and better imputation and coverage of the common and rare allele spectrum, over time, SNP heritability in ASB [antisocial behavior] could approach the family based estimates.”

I don't know why Tielbeek says that, unless he's confusing SNP heritability with PGS: a SNP heritability estimate is unconnected to sample size. Increas... (read more)

How do you plan to deal with the observation that GWASes on personality traits have larger failed, the SNP heritabilities are often near-zero, and that this fits with balancing-selection models of how personality works in humans?

Also, how mature is the concept of Iterated Embryo Selection?

The concept itself dates back to 1998 , as far as I can tell, based on similar ideas dating back at least a decade before that.

There has been enormous progress in various parts of the hypothetical process, like just yesterday Tian et al 2019 reported taking ovarian cells (not eggs) and converting them into mouse eggs and fertilizing and yielding live healthy fertile mice. This is a big step towards 'massive embryo selection' (do 1 egg harvesting cycle, create hundreds or thousands of e... (read more)

One of the amusing things about the 'hinge of history' idea is that some people make the mediocrity argument about their present time - and are wrong.

Isaac Newton, for example, 300 years ago appears to have made an anthropic argument that claims that he lived in a special time which could be considered any kind of, say, 'Revolution', due to the visible acceleration of progress and recent inventions of technologies, were wrong, and in reality, there was an ordinary rate of innovation and the invention of many things recently merely showe... (read more)