All of Greg_Colbourn's Comments + Replies

13 Very Different Stances on AGI

Hi nil :)

They don’t know, esp. about what-it-is-likeness of any sentient experience (although, once again, this may be orthogonal to the risk, at least in theory with unlimited computational power)

Yes, and to to the orthogonality, but I don't think it needs that much computational power (certainly not unlimited). Good enough generalisations could allow it to accomplish a lot (e.g. convincing a lab tech to mix together some mail order proteins/DNA in order to bootstrap nanotech).

or at least to be accurate enough to mislead all of us into a paperclip "hell"

H... (read more)

13 Very Different Stances on AGI

Interesting. Yes I guess such "full-spectrum superintelligence" might well be good by default, but the main worry from the perspective of the Yudkowsky/Bostrom paradigm is not this - perhaps it's better described as super-optimisation, or super-capability (i.e. a blind optimisation process that has no subjective internal experience, and no inclination to gain one, given it's likely initial goals).

Regarding feasibility of conscious AGI / Pearce's full-spectrum superintelligence, maybe it would be possible with biology involved somewhere. But the getting fro... (read more)

2nil8dPearce calls it "full-spectrum" to emphasise the difference w/ Bostrom's "Super-Watson" (using Pearce's words). Given how apparently useful cross-modal world simulations (ie consciousness) have been for evolution, I, again, doubt that such a dumb (in a sense of not knowing what it is doing) process can pose an immediate existential danger to humanity that we won't notice or won't be able to stop. Actually, if I remember correctly, Pearce thinks that if "full-spectrum superintelligence" is going to emerge, it's most likely to be biological, and even post-human (ie it is human descendants who will poses such super minds, not (purely?) silicon-based machines). Pearce sometimes calls this "biotechnological singularity", or "BioSingularity" for short, analogously to Kurzweil's "technological singularity". One can read more about this in Pearce's The Biointelligence Explosion [https://www.hedweb.com/intelligence-explosion/biointelligence-explosion.html] (or in this "extended abstract" [https://www.hedweb.com/intelligence-explosion/index.html]).
13 Very Different Stances on AGI

I will also say that I like Vinding's other work, especially You Are Them. A problem for Alignment is that the AGI isn't Us though (as it's default non-conscious). Perhaps it's possible that an AGI could independently work out Valence Realism and Open/Empty Individualism, and even solve the phenomenal binding problem so as to become conscious itself. But I think these are unlikely possibilities a priori. Although perhaps they should be deliberately aimed for? (Is anyone working on this?)

1nil21d> ... perhaps they should be deliberately aimed for? David Pearce [https://forum.effectivealtruism.org/tag/david-pearce-1] might argue for this if he thought that a "superintelligent" unconscious AGI (implemented on a classical digital computer) were feasible [https://www.physicalism.com/]. E.g. from his The Biointelligence Explosion [https://www.biointelligence-explosion.com/]:
13 Very Different Stances on AGI

I've not read the whole book, but reading the linked article Consciousness – Orthogonal or Crucial?  I feel like Vinding's case is not very convincing. It was written before GPT-3, and this shows. In GPT-3 we already have a (narrow) AI that can convincingly past the Turing Test in writing. Including writing displaying "social skills" and "general wisdom". And very few people are arguing that GPT-3 is conscious.

In general, if you consider that the range of human behaviour is finite, what's to say that it couldn't be recreated simply with a large enough... (read more)

1nil21dHi, Greg :) Thanks for taking your time to read that excerpt and to respond. First of all, the author’s scepticism in a “superintelligent” AGI (as discussed by Bostrom at least) doesn’t rely on consciousness being required for an AGI: i.e. one may think that consciousness is fully orthogonal to intelligence (both in theory and practice) but still on the whole updating away from the AGI risk based on the author’s other arguments from the book. Then, while I do share your scepticism about social skills requiring consciousness (once you have data from conscious people, that is), I do find the author’s points about “general wisdom” (esp. about having phenomenological knowledge) and the science of mind much more convincing (although they are probably much less relevant to the AGI risk). (I won’t repeat the author’s point here: the two corresponding subsections from the piece [https://magnusvinding.com/2020/08/08/consciousness-orthogonal-or-crucial/] are really short to read directly.) Correct me if I’m wrong, but these "social skills" and "general wisdom" are just generalisations (impressive and accurate as they may be) from actual people’s social skills and knowledge. GPT-3 and other ML systems are inherently probabilistic: when they are ~right, they are ~right by accident. They don’t know, esp. about what-it-is-likeness of any sentient experience (although, once again, this may be orthogonal to the risk, at least in theory with unlimited computational power). “Sufficiently” does a lot of work here IMO. Even if something is possible in theory, doesn’t mean it’s going to happen in reality, especially by accident. Also, "... reverse engineer human psychology, hide it’s intentions from us ..." arguably does require a conscious mind, for I don't think (FWIW) that there could be a computationally-feasible substitute (at least one implemented on a classical digital computer) for being conscious in the first place to understand other people (or at least to be accurate e
Convergence thesis between longtermism and neartermism

This especially considering that an all-things-considered (and IMO conservative) estimate for the advent of AGI is 10% chance in (now) 14 years! This is a huge amount of short-term risk! It should not be considered as (exclusively) part of the longtermist cause area.

Convergence thesis between longtermism and neartermism

X-risk as a focus for neartermism

I think it's unfortunate how x-risks are usually lumped in with longtermism , and longtermism is talked about a lot more as a top-level EA cause area these days, and x-risk less so. This considering that, arguably, x-risk is very important from a short-term (or at least medium-term) perspective too. 

As OP says in #2, according to our best estimates, many (most?) people's chances of dying in a global catastrophe over the next 10-25 years are higher than many (any?) other regular causes of death (car accidents, infectiou... (read more)

7michaelchen22dTo be precise, Toby Ord's figure of one in six in ''The Precipice'' refers to the chance of existential catastrophe, not human extinction. Existential catastrophe which includes events such as unrecoverable collapse.
5weeatquince23dGreat point!
2Greg_Colbourn23dThis especially considering that an all-things-considered (and IMO conservative) estimate for the advent of AGI is 10% chance in (now) 14 years [https://www.cold-takes.com/where-ai-forecasting-stands-today/#:~:text=more%20than%20a%2010%25%20chance%20we%27ll%20see%20transformative%20AI%20within%2015%20years%20(by%202036)] ! This is a huge amount of short-term risk! It should not be considered as (exclusively) part of the longtermist cause area.
Why don't governments seem to mind that companies are explicitly trying to make AGIs?

The way I sometimes phrase it to people is that I now think it's more urgent than Climate Change (and people understand that Climate Change is getting quite urgent, and is something that will have a big impact within their lifetimes).

Why don't governments seem to mind that companies are explicitly trying to make AGIs?

Median is ~3-4 decades away. I'd call that "a few", rather than "several" (sorry to nitpick, but I think this is important: several implies "no need to worry about it, probably not going to happen in my lifetime", whereas a few implies (for the majority of people) "this is within my lifetime; I should sit up and pay attention.")

4Greg_Colbourn1moThe way I sometimes phrase it to people is that I now think it's more urgent than Climate Change (and people understand that Climate Change is getting quite urgent, and is something that will have a big impact within their lifetimes).
3Davidmanheim1moFictional evidence! And I haven't seen the movie, but expect it to be far too under-nuanced about how government works.
Why don't governments seem to mind that companies are explicitly trying to make AGIs?

What makes you confident that  "Transformative AI is several decades away"? Holden estimates "more than a 10% chance we'll see transformative AI within 15 years (by 2036)", based on a variety of reports taking different approaches (that are IMO conservative).  Given the magnitude of what is meant by "transformative", governments (and people in general) should really be quite a bit more concerned. As the analogy goes - if you were told that there was a >10% chance of aliens landing on Earth in the next 15 years, then you should really be doing ... (read more)

6HaydnBelfield1moMedian estimate is still decades away. I personally completely agree people should be more concerned.

Governments have trouble responding to things more than a few  years away, and even then, only when it's effectively certain. If they had reliable data that there are aliens showing up in 10 years, I'd expect them to respond by fighting about it and commissioning studies.

My Overview of the AI Alignment Landscape: A Bird’s Eye View

I think AGI would easily be capable of FOOM-ing 100x+ across the board. And as for AGI being developed, it seems like we are getting ever closer with each new breakthrough in ML (and there doesn't seem to be anything fundamentally required that can be said to be "decades away" with high conviction).

My Overview of the AI Alignment Landscape: A Bird’s Eye View

Maybe your view is closer to Eric Drexler's CAIS? That would be a good outcome, but it doesn't seem very likely to be a stable state to me, given that the narrow AIs could be used to speed AGI development. I don't think the world will coordinate around the idea of narrow AIs / CAIS being enough, without a lot of effort around getting people to recognise the dangers of AGI.

1Anthony Repetto1moOh, thank you for showing me his work! As far as I can tell, yes, Comprehensive AI Services seems to be what we are entering already - with GPT-3's Codex [https://openai.com/blog/openai-codex/] writing functioning code a decent percentage of the time, for example! And I agree that limiting AGI would be difficult; I only suppose that it wouldn't hurt us to restrict AGI, assuming that narrow AI does most tasks well. If narrow AI is comparable in performance, (given equal compute) then we wouldn't be missing-out on much, and a competitor who pursues AGI wouldn't see an overwhelming advantage. Playing it safe might be safe. :) And, that would be my argument nudging others to avoid AGI, more than a plea founded on the risks by themselves: "Look how good narrow AI is, already - we probably wouldn't see significant increases in performance from AGI, while AGI would put everyone at risk." If AGI seems 'delicious', then it is more likely to be sought. Yet, if narrow AI is darn-good, AGI becomes less tantalizing. And, for the FOOMing you mentioned in the other thread of replies, one source of algorithmic efficiency is a conversion to symbolic formalism that accurately models the system. Once the over-arching laws are found, modeling can be orders of magnitude faster, rapidly. [e.g. the distribution of tree-size in undisturbed forests always follows a power-law; testing a pair of points on that curve lets you accurately predict all of them!] Yet, such a reduction to symbolic form seems to make the AI's operations much more interpretable, as well as verifiable, and those symbols observed within its neurons by us would not be spoofed. So, I also see developments toward that DNN-to-symbolic bridge as key to BOTH a narrow-AI-powered FOOM, as well as symbolic rigor and verification to protect us. Narrow AI might be used to uncover the equations we would rather rely upon?
My Overview of the AI Alignment Landscape: A Bird’s Eye View

"Neural Descriptor Fields" is promising - their robot learns to grasp from only ten examples

Thanks for these links. Incredible (and scary) progress!

cheaply supply human-brain-scale AI to the nefarious individual

I think we're coming at this from different worldviews. I'm coming from much more of a Yudkowsky/Bostrom perspective, where the thing I worry about is misaligned superintelligent AGI; an existential risk by default. For a ban on AGI to be effective against this, it has to stop every single project reaching AGI. There won't be a stage that lasts any ... (read more)

1Anthony Repetto1moOh, and my apologies for letting questions dangle - I think human intelligence is very limited, in the sense that it is built hyper-redundant against injuries, and so its architecture must be much larger in order to achieve the same task. The latest upgrade to language models, DeepMind's RETRO [https://neurohive.io/en/state-of-the-art/retro-deepmind-language-model/] architecture achieves the same performance as GPT-3 (which is to say, it can write convincing poetry) while using only 1/25th the network. GPT-3 was only 1% of a human brain's connectivity, so RETRO is literally 1/2,500th of a human brain, with human-level performance. I think narrow super-intelligences will dominate, being more efficient than AGI or us. In regards to overall algorithmic efficiency - in only five years we've seen multiple improvements to training and architecture, where what once took a million examples needs ten, or even generalizes to unseen data. Meanwhile, the Lottery Ticket [https://arxiv.org/abs/1803.03635] can make a network 10x smaller, while boosting performance. There was even a supercomputer simulation which neural networks sped 2 BILLION-fold [https://arxiv.org/abs/2001.08055]... which is insane. I expect more jumps in the math ahead, but I don't think we have many of those leaps left before our intelligence-algorithms are just "as good as it gets". Do you see a FOOM-event capable of 10x, 100x, or larger gains left to be found? I would bet there is a 100x is waiting, but it might become tricky and take successively more resources, asymptotic...
1Anthony Repetto1moThank you for diving into this with me :) We might be closer on the meat of the issues than it seems - I sit in the "alignment is exceptionally hard and worthy of consideration" camp, AND I see a nascent FOOM occurring already... yet, I point to narrow superintelligence as the likely mode for profit and success. It seems that narrow AI is already enough to improve itself. (And, the idea that this progress will be lumpy, with diminishing returns sometime soon, is merely my vague forecast informed by general trends of development.) AGI may be attainable at any point X, yet narrow superintelligences may be a better use of those same total resources. More importantly, if narrow AI could do most of the things we want, that tilts my emphasis toward "try our best to stop AGI until we have a long, sober conversation, having seen what tasks are left undone by narrow AI." This is all predicated on my assumption that "narrow AI can self-iterate and fulfill most tasks competently, at lower risk than AGI, and with fewer resources." You could call me a "narrow-minded FOOMist"? :)
My Overview of the AI Alignment Landscape: A Bird’s Eye View

Thanks for the heads up about Hinton's GLOM, Numenta's Sparse Representations and Google's Pathways. The latter in particular seems especially worrying, given Google's resources.

I don't think your arguments regarding Sharpness and Hardness are particularly reassuring though. If an AGI can be made that runs at "real time", what's to stop someone throwing 10, 100, 1000x more compute at it to make it run correspondingly faster? Will they really have spent all the money they have at their disposal on the first prototype? And even if they did, others with more ... (read more)

1Anthony Repetto1moOh, my apologies for not linking to GLOM and such! Hinton's work toward equivariance is particularly interesting because it allows an object to be recognized under myriad permutations and configurations; the recent use of his style of NN in "Neural Descriptor Fields [https://arxiv.org/pdf/2112.05124.pdf]" is promising - their robot learns to grasp from only ten examples [https://yilundu.github.io/ndf/], AND it can grasp even when pose is well-outside the training data - it generalizes! I strongly suspect that we are already seeing the "FOOM," entirely powered by narrow AI. AGI isn't really a pre-requisite to self-improvement: Google used a narrow AI to lay their chips' architecture, for AI-specialized hardware. My hunch is that these narrow AI will be plenty, yet progress will still lurch. Each new improvement is a harder-fought victory, for a diminishing return. Algorithms can't become infinitely better, yet AI has already made 1,000x leaps in various problem-sets ... so I don't expect many more such leaps, ahead. And, in regards to '100x faster brain'... Suppose that an AGI we'd find useful starts at 100 trillion synapses, and for simplicity, we'll call that the 'processing speed' if we run a brain in real-time. "100 trillion synapses-seconds per second" So, if we wanted a brain which was equally competent, yet also running 100x faster, then we would need 100x the computing power, running in parallel to speed operations. That would be 100x more expensive, and if you posit that you had such power on-hand today, then there must have been an earlier date when the amount of compute was only "100 trillion synapses-seconds per second", enough for a real-time brain, only. You can't jump past that earlier date, when only a real-time brain was feasible. You wouldn't wait until you had 100x compute; your first AGI will be real-time, if not slower. GPT-3 and Dall-E are not 'instantaneous', with inference requiring many seconds. So, I expect the same from the first AGI. M
Announcing EffectiveCrypto.org, powered by EA Funds

Some related news: Peter Singer has released a (very limited) NFT series! They're up for auction on OpenSea, with proceeds going to TLYCS.

Mapping of EA

Not exactly what you are looking for, but here is an actual [metaphorical] map (although it could do with updating; it's from Feb 2020):

1Charlie Dougherty2moOh Wow, that's a really fun idea. Thanks for sharing! It's like someone was writing an EA fantasy series.
I’ll pay you a $1,000 bounty for coming up with a good bounty (x-risk related)

I don't think mathematics should be a crux. As I say below, it could be generalised to being offered to anyone a panel of top people in AGI Safety would have on their dream team (who otherwise would be unlikely to work on the problem). Or perhaps “Fields Medalists, Nobel Prize winners in Physics, other equivalent prize recipients in Computer Science, or Philosophy[?], or Economics[?]”. And we could include additional criteria, such as being able to intuit what is being alluded to here. Basically, the idea is to headhunt the very best people for the job, us... (read more)

2WilliamKiely2moThanks, I made a post to try to increase visibility: https://forum.effectivealtruism.org/posts/68drEr2nfLhcJ3mTD/free-usd50-charity-gift-cards-takes-3-minutes-to-claim-one [https://forum.effectivealtruism.org/posts/68drEr2nfLhcJ3mTD/free-usd50-charity-gift-cards-takes-3-minutes-to-claim-one] (It's still available after 3.5+ hours, hopefully will be for several more.)
Greg_Colbourn's Shortform

[Half-baked global health idea based on a conversation with my doctor: earlier cholesterol checks and prescription of statins]

I've recently found out that I've got high (bad) cholesterol, and have been prescribed statins. What surprised me was that my doctor said that they normally wait until the patient has a 10% chance of heart attack or stroke in the next 10 years before they do anything(!) This seems crazy in light of the amount of resources put into preventing things with a similar (or lower) risk profiles, such as Covid, or road traffic accidents. Wo... (read more)

3Josh Jacobson10dThanks for sharing. I'm adding this to my potential research agenda, kept here: https://airtable.com/shrGF5lAwSZpQ7uhP/tblJR9TaKLT41AoSL [https://airtable.com/shrGF5lAwSZpQ7uhP/tblJR9TaKLT41AoSL] and https://airtable.com/shrQdonZuU20cpGR4 [https://airtable.com/shrQdonZuU20cpGR4]
5HaukeHillebrandt2moRomeo Stevens writes about cholesterol here. [https://www.lesswrong.com/posts/PhXENjdXiHhsWGfQo/lifestyle-interventions-to-increase-longevity] Companies like thriva.co offer cheap at home lipid tests. Here are a few recent papers on new drugs: https://academic.oup.com/eurjpc/article/28/11/1279/5898664 [https://academic.oup.com/eurjpc/article/28/11/1279/5898664] https://www.sciencedirect.com/science/article/pii/S0735109721061131?via%3Dihub [https://www.sciencedirect.com/science/article/pii/S0735109721061131?via%3Dihub] Cardiovascular disease is on the rise in emerging economies, so maybe it'd be competitive in the future. Saturated fat seems to be a main culprit: https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD011737.pub3/full [https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD011737.pub3/full] Public health interventions might be a fat tax: https://en.wikipedia.org/wiki/Fat_tax [https://en.wikipedia.org/wiki/Fat_tax] https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD012415.pub2/full [https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD012415.pub2/full] Or donating to the Good Food institute [https://gfi.org/] on human health grounds.
What would you do if you had a lot of money/power/influence and you thought that AI timelines were very short?

I think the main problem is that you don't know for sure that they're close to AGI, or that it  is misaligned, beyond saying that all AGIs are misaligned by default, and what they have looks close to one. If they don't buy this argument -- which I'm assuming they won't given they're otherwise proceeding  -- then you probably won't get very far. 

As for using force (lets assume this is legal/governmental force), we might then find ourselves in a "whack-a-mole" situation, and how do we get global enforcement (/cooperation)?

3acylhalide2moAgreed that you don't know for sure, but "≥ 10% chance of it happening in ≤ 2 years" must have some concrete reasoning backing it. Maybe this reasoning doesn't convince them at first, but it may still be high EV to "explain harder" if you have a personal connection with them. Bring in more expert opinions, appeal to authority - or whatever other mode of reasoning they prefer. You're right that second option is hard and messy. I think it depends on a) are you able to use the code to get any form of strategic advantage without unleashing misaligned AGI b) what form of strategic advantage you get. For instance if you can use the AI to get tech for global surveillance or nanotech for yourself, that would be a good scenario. You can then establish yourself as a world dictator and prevent other attempts at AGI. Maybe don't need to have dictatorial power in all domains, just the ones most revelant to enforcement and the ones that can be built out in very short timeframes. I'm a bit skeptical of govts wielding this power, I figure if you're a billionaire you may want to consider weilding this power yourself. With support of people you trust and/or hire. Worst case the govt takes it from you by force and you're back to where you started. If the US govt takes it by force and is in control of the sole copy of the code, that again becomes a whole scenario to be analysed. There's a lot of scenarios here, hard for one to draw generalised conclusions I guess. Edit: Looks like there's pages on arbitral about it. https://arbital.com/p/pivotal/ [https://arbital.com/p/pivotal/]
What would you do if you had a lot of money/power/influence and you thought that AI timelines were very short?

Imagine it's just the standard AGI scenario where the world ends "by accident", i.e. the people making the AI don't heed the standard risks,  or solve the Control Problem, as outlined in books like Human Compatible and Superintelligence, in a bid to be first to make AGI  (perhaps for economic incentives, or perhaps for your ** scenario). I imagine it will also be hard to know who exactly the actors are, but you could have some ideas (e.g. the leading AI companies, certain governments etc). 

1acylhalide2moOkay. I'm still gonna assume that they have atleast read some AI alignment theory. Then I think some options are: * convincing them they are close to AGI / convinve them their AGI is misaligned, whichever of the two is important. * getting them stopped by force Convincing them requires opening dialogue with them, not being hostile and convincing them that you're atleast somewhat aligned with them. Money might or might not help with this. Getting them to stop could mean appealing to the public, or funding a militia that enters their facility by force. Appealing to the govt and public is too slow unless the govt already has people very aware of the problem. It can work though. If you do use force and manage to steal their code, one desperate option is to attempt getting a powerful but not superintelligent AI that gives you personally or the US govt or anyone else trustworthy a decisive strategic advantage to prevent future people from working on it - burn all copies etc.
I’ll pay you a $1,000 bounty for coming up with a good bounty (x-risk related)

Good idea about the fellowship. I've been thinking that it would need to come from somewhere prestigious. Perhaps CHAI, FLI or CSER, or a combination of such academic institutions? If it was from, say, a lone crypto millionaire, they might risk being dismissed as a crackpot, and by extension risk damaging the reputation of AGI Safety. Then again, perhaps the amounts of money just make it too outrageous to fly in academic circles? Maybe we should be looking to something like sports or entertainment instead? Compare the salary to that of e.g. top footballers... (read more)

Discussion with Eliezer Yudkowsky on AGI interventions

Yes, concern is optimisation during training. My intuition is along the lines of "sufficiently large pile of linear algebra with reward function-> basic AI drives maximise reward->reverse engineers [human behaviour / protein folding / etc] and manipulates the world so as to maximise it's reward ->[foom / doom]".

I wouldn't say "personality" comes into it. In the above scenario the giant pile of linear algebra is completely unconscious and lacks self-awareness; it's more akin to a force of nature, a blind optimisation process.

3Brian_Tomasik2moThanks. :) Regarding the AGI's "personality", what I meant was what the AGI itself wants to do, if we imagine it to be like a person, rather than what the training that produced it was optimizing for. If we think of gradient descent to train the AGI as like evolution and the AGI at some step of training as like a particular human in humanity's evolution, then while evolution itself is optimizing something, the individual human is just an adaptation executor and doesn't directly care about his inclusive fitness. He just responds to his environment as he was programmed to do. Likewise, the GPT-X agent may not really care about trying to reduce training errors by modifying its network weights; it just responds to its inputs in human-ish ways.
Discussion with Eliezer Yudkowsky on AGI interventions

It explains how a GPT-X could become an AGI (via world modelling). I think then things like the basic drives would take over. However, maybe it's not the end result model that we should be looking at as dangerous, but rather the training process? A ML-based (proto-)AGI could do all sorts of dangerous (consequentialist, basic-AI-drives-y) things whilst trying to optimise for performance in training.

1Brian_Tomasik2moWhere would those basic drives come from (apart from during training)? An already trained GPT-3 model tries to output text that looks human-like, so we might imagine that a GPT-X AGI would also try to behave in ways that look human-like, and most humans aren't very consequentialist. Humans do try to preserve themselves against harm or death, but not in an "I need to take over the world to ensure I'm not killed" kind of way. If your concern is about optimization during training, that makes sense, though I'm confused as to whether it's dangerous if the AI only updates its weights via a human-specified gradient-descent process, and the AI's "personality" doesn't care about how accurate its output is.
How many people should get self-study grants and how can we find them?


My original idea (quote below) included funding people at equivalent costs remotely. Basically no one asked about that. I guess because not many EAs have that low a living cost (~£6k/yr). And not that many could without moving to a different town (or country), and there isn't much appetite for that / coordination is difficult. 

Maybe we need a grant specifically for people to work on research remotely that has a higher living cost cap? Or a hierarchy of such grants with a range of living costs that are proportionally harder to get the higher the costs ... (read more)

2casebash2moYeah, a hierarchy of grants would make sense.
Discussion with Eliezer Yudkowsky on AGI interventions

Here is an argument for how GPT-X might lead to proto-AGI in a more concrete, human-aided, way: 

..language modelling has one crucial difference from Chess or Go or image classification. Natural language essentially encodes information about the world—the entire world, not just the world of the Goban, in a much more expressive way than any other modality ever could.[1] By harnessing the world model embedded in the language model, it may be possible to build a proto-AGI.

...

This is more a thought experiment than something that’s actually going to happen

... (read more)
1Brian_Tomasik2moThanks. :) Is that just a general note along the lines of what I was saying, or does it explain how a GPT-X AGI would become consequentialist?
Discussion with Eliezer Yudkowsky on AGI interventions

Steve Omohundro 

...Google and others are using Mixture-of-Experts to avoid some of that cost: https://arxiv.org/abs/1701.06538

Matrix multiply is a pretty inefficient primitive and alternatives are being explored: https://arxiv.org/abs/2106.10860

These stand out for me as causes for alarm. Anything that makes ML significantly more efficient as an AI paradigm seems like it shortens timelines. Can anyone say why they aren't cause for alarm? (See also)

Discussion with Eliezer Yudkowsky on AGI interventions

Eliezer Yudkowsky 

...Throwing more money at this problem does not obviously help because it just produces more low-quality work

Maybe you're not thinking big enough? How about offering the world's best mathematicians  (e.g. Terence Tao) a lot of money to work on AGI Safety. Say $5M to work on the problem for a year. Perhaps have it open to any Fields Medal recipient. (More)
 

Discussion with Eliezer Yudkowsky on AGI interventions

I think the issue is more along the lines of the superhuman-but-not-exceedingly-superhuman AGI quickly becoming an exceedingly-superhuman AGI (i.e. a Superintelligence) via recursive self-improvement (imagine a genius being able to think 10 times faster, then use that time to make itself think 1000 times faster, etc). And AGIs should tend toward consequentialism via convergent instrumental goals (e.g.).

Or are you saying that you expect the superhuman France/China/Facebook AGI to remain boxed?

2Brian_Tomasik2moI guess it depends how superhuman we're imagining the AGI to be, but if it was merely as intelligent as like 100 human AGI experts, it wouldn't necessarily speed up AGI progress enormously? Plus, it would need lots of compute to run, presumably on specialized hardware, so I'm not sure it could expand itself that far without being allowed to? Perhaps its best strategy would be to play nice for the time being so that humans would voluntarily give it more compute and control over the world. Hm, if an agent is consequentialist, then it will have convergent instrumental subgoals. But what if the agent isn't consequentialist to begin with? For example, if we imagine that GPT-7 is human-level AGI, this AGI might have human-type common sense. If you asked it to get you coffee, it might try to do so in a somewhat common-sense way, without scheming about how to take over the world in the process, because humans usually don't scheme about taking over the world or preserving their utility functions at all costs? But I don't know if that's right; I wonder what AI-safety experts think. Also, GPT-type AIs still seem very tricky to control, but for now that's because their behavior is weird and unpredictable rather than because they're scheming consequentialists.
Discussion with Eliezer Yudkowsky on AGI interventions

Has anyone tried GPT3-ing this to see if it comes up with any interesting ideas?

Discussion with Eliezer Yudkowsky on AGI interventions

Deepmind would have lots of penalty-free affordance internally for people to not publish things, and to work in internal partitions that didn't spread their ideas to all the rest of Deepmind.

Companies like Apple and Dyson operate like this (keeping their IP tightly under wraps right up until products are launched). Maybe they could be useful recruiting grounds?

Make a $100 donation into $200 (or more)

CEEALAR now has it's own page, separate from PPF (our US fiscal sponsor, which donations are still routed through). The benefit of this is that donations can be matched separately (with the PPF fundraisers, you can only get a match for one of them).

4WilliamKiely2moGreat, just donated!
How many people should get self-study grants and how can we find them?

(Or, if it's better for EA to stay smaller, then more dedicated self-starting __s should be funded to work on __ full-time.)

How many people should get self-study grants and how can we find them?

I imagine A < B in terms of numbers of people; and B ≈ C, given you are pre-selecting for "self-starting EAs". I think just being dedicated enough to EA to want to spend a year working on it full time is a reasonably strong signal that you would have something to contribute, given that dedication seems to require a strong understanding in the case of EA. And self-starting + dedication + a strong understanding + working full-time on trying to have impact at the margin should = impact at the margin.

Obviously there is then the important detail of how big t... (read more)

7Linch2moI think being dedicated enough to EA to want to spend a year working on it full time =/= being dedicated enough to EA to actually work on it full-time with minimal management. I agree the latter is a pretty strong signal, especially if you're able to identify important things to work on. Holden's post on career choice for longtermists [https://forum.effectivealtruism.org/posts/bud2ssJLQ33pSemKH/my-current-impressions-on-career-choice-for-longtermists#On_track_2] say that this is an important milestone for researchers:
2Greg_Colbourn2mo(Or, if it's better for EA to stay smaller, then more dedicated self-starting __ [https://forum.effectivealtruism.org/posts/QDWoBgDKpRrG8u76y/what-high-level-change-would-you-make-to-ea-strategy?commentId=ggfWKwXHwFGBH3dak] s should be funded to work on __ full-time.)
Forecasting transformative AI: the "biological anchors" method in a nutshell

with the exception of "mixture-of-experts models" that I think we should disregard for these purposes, for reasons I won't go into here

This is taken from a footnote. Clicking on the link and reading the abstract, it immediately jumped out as something that we should be potentially quite concerned about (i.e. the potential to scale models by ~1000x using the same compute!), so I'm curious about the reasons for disregarding that you didn't go into in the post. Can you go into them here?

Using the "Cited by" feature on Google Scholar, I've found some more rece... (read more)

3gwern19dYes, the brain is sparse and semi-modularized, but it'd be hard to really call it more 'brain-like' than dense models. Brains have all sorts of very long range connections in a small-world topology, where most of the connections may be local but there's still connections to distant parts, and those are important; distant brain regions can also communicate and be swapped in and out as the brain recurs and ponders. The current breed of MoEs along the lines of Switch Transformer don't do any of that. They do a single pass, and each module is completely local and firewalled from the others. This is what makes them so 'efficient': they are so separate they can be run and optimized easily in parallel with no communication and they handle only limited parts of the problem so they are still early in the scaling curve. To continue Holden's analogy, it's not so much like gluing 100 mouse brains together (or in my expression, 'gluing a bunch of chihuahuas back to back and expecting them to hunt like a wolf'), it's like having one mouse brain as a harried overworked MBA manager who must send an email off to one or two of his 99 mouse employees, each of whom then must take care of the job entirely on their own that instant (and are not allowed to communicate or ask for clarification or delegate to any of the other mice). The more you add recurrency or flexible composition of experts or long-range connections, the more you give up what made them cheap in the first place... I continue to be skeptical that MoEs as currently pursued are anything but a distracting pennywise-poundfoolish sort of diversion, settling for trying to ape GPT-3 at mere fractional savings. Sure, approaches like ERNIE 3.0 Titan [https://arxiv.org/abs/2112.12731#baidu] look horrifically expensive, but at least they look like they're pushing into new territory.
4kokotajlod2moMy impression from following r/mlscaling for a while and reading a bunch of comments by Gwern and also various papers... is that MoE models aren't that good. But I don't really have a good understanding of this so I could be wrong. I guess I just am deferring to others like Gwern. I also am thinking if MoE models were anywhere near as good as 1000x then we would have seen something dramatically better than GPT-3 already and we haven't.
How to make the best of the most important century?

Spreading ideas and building communities.

Holden, have you considered hosting seminars on the Most Important Century? (And incentivising important people to attend?) I've outlined this idea here.

I’ll pay you a $1,000 bounty for coming up with a good bounty (x-risk related)

(Sorry if some of my ideas are fairy big budget, but EA seems to have quite a big budget these days)

I’ll pay you a $1,000 bounty for coming up with a good bounty (x-risk related)

Maybe if such a thing is successfully pulled off, it could be edited into a documentary TV series, say with clips from each week's discussion taken from each of the groups, and an overarching narrative in interludes with music, graphics, stock footage (plane on runway) etc.

3Greg_Colbourn3mo(Sorry if some of my ideas are fairy big budget, but EA seems to have quite [https://forum.effectivealtruism.org/posts/zA6AnNnYBwuokF8kB/is-effective-altruism-growing-an-update-on-the-stock-of#How_many_funds_are_committed_to_effective_altruism_] a [https://forum.effectivealtruism.org/posts/nXL2MeQQBoHknpz8X/what-s-the-role-of-donations-now-that-the-ea-movement-is] big [https://www.facebook.com/groups/effective.altruists/posts/4537496452973343/] budget [https://forum.effectivealtruism.org/posts/ckcoSe3CS2n3BW3aT/what-ea-projects-could-grow-to-become-megaprojects#o2BzEMLtYJ4TZGQdE] these days)
I’ll pay you a $1,000 bounty for coming up with a good bounty (x-risk related)

Note there have already been Most Important Century seminars hosted. I missed this one. Would be interested to hear how it went.

I’ll pay you a $1,000 bounty for coming up with a good bounty (x-risk related)

Are there any prior examples of this kind of thing? (I haven't found any with a quick search.)

I’ll pay you a $1,000 bounty for coming up with a good bounty (x-risk related)

Introduce important people* to the most important ideas by way of having seminars they are paid to attend. I recommend Holden Karnofsky’s Most Important Century series for this as it is highly engaging, very readable, and has many jumping off points to go into more depth; but other things are also very good. The format could be groups of 4, with a moderator (and optionally the authors of the pieces under discussion on the side lines to offer clarity / answer questions). It could be livestreamed for accountability (on various platforms to diversify audience... (read more)

2Greg_Colbourn3moMaybe if such a thing is successfully pulled off, it could be edited into a documentary TV series, say with clips from each week's discussion taken from each of the groups, and an overarching narrative in interludes with music, graphics, stock footage (plane on runway [https://www.cold-takes.com/call-to-vigilance/]) etc.
2Greg_Colbourn3moNote there have already been Most Important Century seminars hosted. I missed this one [https://forum.effectivealtruism.org/posts/JWxGruyQKNxnZiu9n/a-discussion-of-holden-karnofsky-s-most-important-century] . Would be interested to hear how it went.
2Greg_Colbourn3moAre there any prior examples of this kind of thing? (I haven't found any with a quick search.)
EA Infrastructure Fund: Ask us anything!

"can we build a structure that allows separation between, and controlled flow of talent and other resources, different subcommunities?"

Interesting discussion. What if there was a separate brand for a mass movement version of EA?

What high-level change would you make to EA strategy?

I've sometimes wondered whether it would be good for there to be a distinct brand and movement for less hardcore EA, that is less concerned with prestige, less elitist, more relaxed, and with more mainstream appeal. Perhaps it could be thought of as the Championship to EA's Premier League. I think there are already examples, e.g. Probably Good (alternative to 80,000 Hours), TLYCS and OFTW (alternatives to GWWC), and the different tiers of EA investing groups (rough and ready vs careful and considered). Places where you feel comfortable only spending 5 minu... (read more)

2Greg_Colbourn3moSome related discussion here [https://forum.effectivealtruism.org/posts/KesWktndWZfGcBbHZ/ea-infrastructure-fund-ask-us-anything?commentId=yXcHQWoHEYdZdrNvr#yXcHQWoHEYdZdrNvr] .
List of EA-related organisations

Giving the EA funding saturation situation, it would be great to see this list include room for more funding (RFMF), or link to a spreadsheet similar to GiveWell's (that Ben Todd tweeted about), but for all EA related orgs.

I’ll pay you a $1,000 bounty for coming up with a good bounty (x-risk related)

True. But maybe the limiting factor is just the consideration of such ideas as a possibility? When I was growing up, I wanted to be a scientist, liked space-themed Sci-Fi, and cared about many issues in the world (e.g. climate change, human rights); but I didn't care about having or wanting money (in fact I mostly thought it was crass), or really think much about it as a means to achieving ends relating to my interests. It wasn't until reading about (proto-)EA ideas that it clicked.

I’ll pay you a $1,000 bounty for coming up with a good bounty (x-risk related)

Interesting. I wonder: many people say they aren't motivated by money, but how many of them have seriously considered what they could do with it other than personal consumption? And how many have actually been offered a lot of money -- to do something different to what they would otherwise do, that isn't immoral or illegal -- and turned it down? What if it was a hundred million, or a billion dollars? Or, what if the time commitment was lower - say 6 months, or 3 months?

2Davidmanheim3moGood point. And yes, it seems likely that they'd change their research, but I don't think that motivation and curiosity are as transferable. Still on net not a bad idea, but I'm still skeptical it would be this easy.
2Linch3moIf top mathematicians had an EA mindset towards money, they would most likely not be publishing pure math papers.
Liberty in North Korea, quick cost-effectiveness estimate

Note that LINK is on every.org, so for a short time you can get a donation up to $100 matched (and double your cost-effectiveness estimate).

Load More