AGI rising: why we are in a new era of acute risk and increasing public awareness, and what to do now

Greg_Colbourn ⏸️

AGI rising: why we are in a new era of acute risk and increasing public awareness, and what to do now

Greg_Colbourn ⏸️

15 min readMay 2, 2023

Comments 35

Sorted by

New & upvoted

Magnus Vinding

To push back a bit on the fast software-driven takeoff (i.e. a fast takeoff driven primarily by innovations in software):

Common objections to this narrative [of a fast software-driven takeoff] are that there won’t be enough compute, or data, for this to happen. These don’t hold water after a cursory examination of our situation. We are nowhere near close to the physical limits to computation ...

While we're nowhere near the physical limits to computation, it's still true that hardware progress has slowed down considerably on various measures. I think the steelman of the compute-based argument against a fast software-driven takeoff is not that the ultimate limits to computation are near, but rather that the pace of hardware progress is unlikely to be explosively fast (e.g. in light of recent trends that arguably point in the opposite direction, and because software progress per se seems insufficient for driving explosive hardware progress).

We're on ~10^18 FLOPS for large clusters at the moment^[14], but there are likely enough GPUs available for 100 times this^[15]. And the cost of this – ~$10B - is affordable for many large tech companies and national governments, and even individuals!

That actors can afford to create this next generation of AIs does not imply that those AIs will in turn lead to a hard takeoff in capabilities. From my perspective at least, that seems like an unargued assumption here.

Data is not a show-stopper either. Sure, ~all the text on the internet might’ve already been digested, but Google could readily record more words per day via phone mics^[16] than the number used to train GPT-4^[17]. These may or may not be as high quality as text, but 1000x as many as all the text on the internet could be gathered within months^[18]. Then there are all the billions of high-res video cameras (phones and CCTV), and sensors in the world^[19]. And if that is not enough, there is already a fast-growing synthetic data industry serving the ML community’s ever growing thirst for data to train their models on.

A key question is whether this extra data would be all that valuable to the main tasks of concern. For example, it seems unclear whether low-quality data from phone conversations, video cameras, etc. would give that much of a boost to a model's ability to write code. So I don't think the point made above, as it stands, is a strong rebuttal to the claim that data will soon be a limiting bottleneck to significant capability gains. (Some related posts.)

The time for talking politely to (and working with) big AI is over.

This is another claim I would push back against. For instance, from a perspective concerned with the reduction of s-risks, one could argue that talking politely to, and working with, leading AI companies is in fact the most responsible thing to do, and that taking a less cooperative stance is unduly risky and irresponsible. To be clear, I'm not saying that this is obviously the case, but I'm trying to say that it's not clear-cut either way. Good arguments can be made for a different approach, and this seems true for a wide range of altruistic values.

Tomas B.

Current scaling "laws" are not laws of nature. And there are already worrying signs that things like dataset optimization/pruning, curriculum learning and synthetic data might well break them - It seems likely to me that LLMs will be useful in all three. I would still be worried even if LLMs prove useless in enhancing architecture search.

JakubK

Current scaling "laws" are not laws of nature. And there are already worrying signs that things like dataset optimization/pruning, curriculum learning and synthetic data might well break them

Interesting -- can you provide some citations?

Peter Slattery 🔸

Thanks for writing this - it was useful to read the pushbacks!

As I said below, I want more synthesis of these sorts of arguments. I know that some academic groups are preparing literature reviews of the key arguments for and against AGI risk.

I really think that we should be doing that for ourselves as a community and to make sure that we are able to present busy smart people with more compelling content than a range of arguments spread across many different forum posts.

I don't think that that is going to cut it for many people in the policy space.

Greg_Colbourn ⏸️

Agree. But at the same time, we need to do this fast! The typical academic paper review cycle is far too slow for this. We probably need groups like SAGE (and Independent SAGE?) to step in. In fact, I'll try and get hold of them.. (they are for "emergencies" in general, not just Covid^[1])

^{^}
Although it looks like they are highly specialised on viral threats. They would need totally new teams to be formed for AI. Maybe Hinton should chair?

Greg_Colbourn ⏸️

hardware progress has slowed down considerably on various measures

I don't think this matters, as per the next point about there already being enough compute for doom [Edit: I've relegated the "nowhere near close to the physical limits to computation" sentence to a footnote and added Magnus' reference on slowdown to it].

That actors can afford to create this next generation of AIs does not imply that those AIs will in turn lead to a hard takeoff in capabilities. From my perspective at least, that seems like an unargued assumption here.

I think the burden of proof here needs to shift to those willing to gamble on the safety of 100x larger systems. All I'm really saying here is that the risk is way too high for comfort (given the jumps in capabilities we've seen so far going from GPT-3->GPT3.5->GPT-4).

[Meta: would appreciate separate points being made in separate comments].
Will look into your links re data and respond later.

from a perspective concerned with the reduction of s-risks, one could argue that talking politely to, and working with, leading AI companies is in fact the most responsible thing to do, and that taking a less cooperative stance is unduly risky and irresponsible.

I'm not sure what you are saying here? Do you think there is a risk of AI companies deliberately causing s-risks (e.g. releasing a basilisk) if we don't play nice!? They may be crazy in a sense of being reckless with the fate of billions of people's lives, but I don't think they are that crazy (in a sense of being sadistically malicious and spiteful toward their opponents)!

Magnus Vinding

I'm not sure what you are saying here? Do you think there is a risk of AI companies deliberately causing s-risks (e.g. releasing a basilisk) if we don't play nice!?

No, I didn't mean anything like that (although such crazy unlikely risks might also be marginally better reduced through cooperation with these actors). I was simply suggesting that cooperation could be a more effective way to reduce risks of worst-case outcomes that might occur in the absence of cooperative work to prevent them, i.e. work of the directional kind gestured at in my other comment (e.g. because ensuring the inclusion of certain measures to avoid worst-case outcomes has higher EV than does work to slow down AI). Again, I'm not saying that this is definitely the case, but it could well be. It's fairly unclear, in my view.

Greg_Colbourn ⏸️

Ok. I don't put much weight on s-risks being a likely outcome. Far more likely seems to be just that the solar system (and beyond) will be arranged in some (to us) arbitrary way, and all carbon-based life will be lost as collateral damage.

Although I guess if you are looking a bit nearer term, then s-risk from misuse could be quite high. But I don't think any of the major players (OpenAI, Deepmind, Anthropic) are even really working on trying to prevent misuse at all as part of their strategy (their core AI Alignment work is on aligning the AIs, rather than the humans using them!) So actually, this is just another reason to shut it all down.

Magnus Vinding

Thanks for your reply, Greg :)

I don't think this matters, as per the next point about there already being enough compute for doom

That is what I did not find adequately justified or argued for in the post.

I think the burden of proof here needs to shift to those willing to gamble on the safety of 100x larger systems.

I suspect that a different framing might be more realistic and more apt from our perspective. In terms of helpful actions we can take, I more see the choice before us as one between trying to slow down development vs. trying to steer future development in better (or less bad) directions conditional on the current pace of development continuing (of course, one could dedicate resources to both, but one would still need to prioritize between them). Both of those choices (as well as graded allocations between them) seem to come with a lot of risks, and they both strike me as gambles with potentially serious downsides. I don't think there's really a "safe" choice here.

All I'm really saying here is that the risk is way too high for comfort

I'd agree with that, but that seems different from saying that a fast software-driven takeoff is the most likely scenario, or that trying to slow down development is the most important or effective thing to do (e.g. compared to the alternative option mentioned above).

Greg_Colbourn ⏸️

both strike me as gambles with potentially serious downsides.

What are the downsides from slowing down? Things like not curing diseases and ageing? Eliminating wild animal suffering? I address that here: "it’s a rather depressing thought. We may be far closer to the Dune universe than the Culture one (the worry driving a future Butlerian Jihad will be the advancement of AGI algorithms to the point of individual laptops and phones being able to end the world). For those who may worry about the loss of the “glorious transhumanist future”, and in particular, radical life extension and cryonic reanimation (I’m in favour of these things), I think there is some consolation in thinking that if a really strong taboo emerges around AGI, to the point of stopping all algorithm advancement, we can still achieve these ends using standard supercomputers, bioinformatics and human scientists. I hope so."

To be clear, I'll also say that it's far too late to only steer future development better. For that, Alignment needs to be 10 years ahead of where it is now!

a fast software-driven takeoff is the most likely scenario

I don't think you need to believe this to want to be slamming on the brakes now. As mentioned in the OP, is the prospect of mere imminent global catastrophe not enough?

Magnus Vinding

What are the downsides from slowing down?

I'd again prefer to frame the issue as "what are the downsides from spending marginal resources on efforts to slow down?" I think the main downside, from this marginal perspective, is opportunity costs in terms of other efforts to reduce future risks, e.g. trying to implement "fail-safe measures"/"separation from hyperexistential risk" in case a slowdown is insufficiently likely to be successful. There are various ideas that one could try to implement.

In other words, a serious downside of betting chiefly on efforts to slow down over these alternative options could be that these s-risks/hyperexistential risks would end up being significantly greater in counterfactual terms (again, not saying this is clearly the case, but, FWIW, I doubt that efforts to slow down are among the most effective ways to reduce risks like these).

a fast software-driven takeoff is the most likely scenario
I don't think you need to believe this to want to be slamming on the breaks on now.

Didn't mean to say that that's a necessary condition for wanting to slow down. But again, I still think it's highly unclear whether efforts that push for slower progress are more beneficial than alternative efforts.

Greg_Colbourn ⏸️

I think it's a very hard sell to try and get people to sacrifice themselves (and the whole world) for the sake of preventing "fates worse than death". At that point most people would probably just be pretty nihilistic. It also feels like it's not far off basically just giving up hope: the future is, at best, non-existence for sentient life; but we should still focus our efforts on avoiding hell. Nope. We should be doing all we can now to avoid having to face such a predicament! Global moratorium on AGI, now.

Magnus Vinding

I think it's a very hard sell to try and get people to sacrifice themselves (and the whole world) for the sake of preventing "fates worse than death".

I'm not talking about people sacrificing themselves or the whole world. Even if we were to adopt a purely survivalist perspective, I think it's still far from obvious that trying to slow things down is more effective than is focusing on other aims. After all, the space of alternative aims that one could focus on is vast, and trying to slow things down comes with non-trivial risks of its own (e.g. risks of backlash from tech-accelerationists). Again, I'm not saying it's clear; I'm saying that it seems to me unclear either way.

We should be doing all we can now to avoid having to face such a predicament!

But, as I see it, what's at issue is what the best way is to avoid such a predicament/how to best navigate given our current all-too risky predicament.

FWIW, I think that a lot of the discussion around this issue appears strongly fear-driven, to such an extent that it seems to get in the way of sober and helpful analysis. This is, to be sure, extremely understandable. But I also suspect that it is not the optimal way to figure out how to best achieve our aims, nor an effective way to persuade readers on this forum. Likewise, I suspect that rallying calls along the lines of "Global moratorium on AGI, now" might generally be received less well than would, say, a deeper analysis of the reasons for and against attempts to institute that policy.

Greg_Colbourn ⏸️

I feel like I'm one of the main characters in the film Don't Look Up here.

the space of alternative aims that one could focus on is vast

Please can you name 10? The way I see it is - either alignment is solved in time with business as usual^[1], or we Pause to allow time for alignment to be solved (or establish it's impossibility). It is not a complicated situation. No need to be worrying about "fates worse than death" at this juncture.

^{^}
seems highly unlikely, but please say if you think there are promising solutions here

Matt Brooks

Do you not trust Ilya when he says they have plenty more data?

https://youtu.be/Yf1o0TQzry8?t=656

Magnus Vinding

I didn't claim that there isn't plenty more data. But a relevant question is: plenty more data for what? He says that the data situation looks pretty good, which I trust is true in many domains (e.g. video data), and that data would probably in turn improve performance in those domains. But I don't see him claiming that the data situation looks good in terms of ensuring significant performance gains across all domains, which would be a more specific and stronger claim.

Moreover, the deference question could be posed in the other direction as well, e.g. do you not trust the careful data collection and projections of Epoch? (Though again, Ilya saying that the data situation looks pretty good is arguably not in conflict with Epoch's projections — nor with any claim I made above — mostly because his brief "pretty good" remark is quite vague.)

Note also that, at least in some domains, OpenAI could end up having less data to train their models with going forward, as they might have been using data illegally.

Greg_Colbourn ⏸️

Let's hope that OpenAI is forced to pull GPT-4 over the illegal data harvesting used to create it.

Greg_Colbourn ⏸️

Coming back to the point about data. Whilst Epoch gathered some data showing that the stock high quality text data might soon be exhausted, their overall conclusion is that there is only a “20% chance that the scaling (as measured in training compute) of ML models will significantly slow down by 2040 due to a lack of training data.”. Regarding Jacob Buckman's point about chess, he actually outlines a way around that (training data provided by narrow AI). As a counter to the wider point about the need for active learning, see DeepMind's Adaptive Agent and the Voyager "lifelong learning" Minecraft agent, both of which seem like impressive steps in this direction.

Peter Slattery 🔸

Thanks for writing this. I appreciate the effort and sentiment. My quick and unpolished thoughts are below. I wrote this very quickly, so feel free to critique.

The TLDR is that I think that this is good with some caveats but also that we need more work on our ecosystem to be able to do outreach (and everything else) better.

I think we need a better AI Safety movement to properly do and benefit from outreach work. Otherwise, this and similar posts for outreach/action are somewhat like a call to arms without the strategy, weapons and logistics structure needed to support them.

Doing the things you mention is probably better than doing nothing (some of these more than others), but it's far what is possible in terms or risk minimisation and expected impact.

What do we need for the AI Safety movement to properly do and benefit from outreach work?

I think that doing effective collective outreach will require us to be more centralised and coordinated.

Right now, we have people like you who seem to believe that we need to act urgently to engage people and raise awareness, in opposition to other influential people like Rohin Shah, Oliver Harbynka, who seem to oppose movement building (though this may just be the recruitment element).

The polarisation and uncertainty promotes inaction.

I therefore don't think that we will get anything close to maximally effective awareness raising about AI risk until we have a related strategy and operational plan that has enough support from key stakeholders or is led by one key stakeholder (e.g., Will/Holden/Paul) and actioned by those who trust that person's takes.

Here are my related (low confidence) intuitions (based on this and related conversations mainly) for what to do next:

We need to find/fund/choose some/more people/process to drive overall strategy and operation for the mainstream AI Safety community. For instance, we could just have some sort of survey/voting system to capture community preferences/elect someone. I don't know what makes sense now, but it's worth thinking about.

When we know what the community/representatives see as the strategy and supporting operation, we need someone/some process to figure out who is responsible for executing the overall strategy and parts of the operations and communicating them to relevant people. We need behaviour level statements for 'who needs to do what differently'.

When we know 'who needs to do what differently' we need to determine and address the blockers and enablers to scale and sustain the strategy and operation (e.g., we likely need researchers to find what communication works with different audiences, communicators to write things, connect with, and win over, influential/powerful people, recruiters to recruit the human resources, developers and designers to make persuasive digital media, managers to manage these groups, entrepreneurs to start and scale the project, and funders to support the whole thing etc).

It's a big ask, but it might be our best shot.

Why hasn't somebody done this already?

As I see it, the main reason for all of the above is a lack of shared language and understanding which merged because of how the AI safety community developed.

Movement building/field building mean different things to different people and no-one knows what the community collectively support or oppose in this regard. This uncertainty reduces attempts to do anything on behalf of the community or the chances of success if anyone tries.

Perhaps because of this no-one who could curate preferences and set a direction (e.g., Will/Holden/Paul) feels confident to do so.

It's potentially a chicken and egg or coincidence of wants problem where most people would like someone like Holden to drive the agenda, but he doesn't know or thinks someone would be better suited (and they don’t know). Or the people who could lead somehow know that the community doesn’t want anyone to lead it in this way, but haven't communicated this, so I don’t know that yet.

What happens if we keep going as we are?

I think that the EA community (with some exceptions) will mostly continue to function like a decentralised group of activists, posting conflicting opinions in different forums and social media channels, while doing high quality, but small scale, AI safety governance, technical and strategy work that is mostly known and respected in the communities it is produced in.

Various other more centralised groups with leaders like Sam Altman, Tristan Harris, Tina Gebru etc will drive the conversations and changes. That might be for the best, but I suspect not.

Urgent, unplanned communication by EA acting insolation poses many risks. If lots of people who don't know what works for changing people's minds and behaviours post lots of things about how they feel this could be bad.

These people could very well end up in isolated communities (e.g., just like many vegan activists I see who are mainly just reaching vegan followers on social media).

They could poison the well and make people associate AI safety with poorly informed and overconfident pessimists.

If people engage in civil disobedience we could end being feared and hated and subsequently excluded from consideration and conversation.

Our actions could create abiding associations that will damage later attempts to persuade by more persuasive sources.

This could be the unilateralist's curse brought to life.

Other thoughts/suggestions

Test the communication in small scale (e.g., with a small sample of people on mechanical turk or with friends) before you do large scale outreach

Think about taking steps back to prioritise between the behaviour to rule out the ones with more downside risk (so better to write letters to representatives than posts to large audiences on social media if unsure what is persuasive).

Don’t do civil disobedience unless you have read the literature about when and where it works (and maybe just don’t do it - that could backfire badly).

Think about the AI Safety ecosystem and indirect ways to get more of what you want by influencing/aiding people or processes within it:

For instance, I'd like for progress on questions like:

- what are the main arguments for and against doing certain things (e.g., the AI pause/public awareness raising), what is the expert consensus on whether a strategy/action would be a good idea or not (e.g., what do superforcasters/AI orgs recommend)?

- When we have evidence for a strategy/action, then: Who needs to do what differently? Who do we need to communicate to, and what do we know is persuasive to them/how can we test that?

- Which current AI safety (e.g., technical, strategy, movement building) projects are the ones that are worth prioritising the allocation of resources (funding, time, advocacy) to etc? What do experts think?
- What voices/messages can we amplify when we communicate? It's much easier to share something good from an expert than write it.

- Who could work with others for mutual benefit but doesn’t realise it yet?

I am thinking about, and doing, a little of some of these things, but have other obligations for 3-6 months and some uncertainty about whether I am well suited to do them.

Greg_Colbourn ⏸️

These all seem like good suggestions, if we still had years. But what if we really do only have months (to get a global AGI moratorium in place)? In some sense the "fog of war" may already be upon us (there are already too many further things for me to read and synthesise, and analysis paralysis seems like a great path toward death). How did action on Covid unfold? Did all these kinds of things happen first before we got to lockdowns?

vegan activists

This is quite different. It's about personal survival of each and every person on Earth, and their families. (No concern for other people or animals is needed!)

This could be the unilateralist's curse brought to life.

Could it possibly be worse than what's already happened with DeepMind, OpenAI and Anthropic (and now Musk's X.ai)?

have other obligations for 3-6 months

Is there any way you can get out of those other obligations? Time really is of the essence here. Things are already moving fast, whether EA/the AI Safety community is coordinated and onboard or not.

Geoffrey Miller

Peter -- good post; these all seem reasonable as comments.

However, let me offer a counter-point, based on my pretty active engagement on Twitter about AI X-risk over the last few weeks: it's often very hard to predict which public outreach strategies, messages, memes, and points will resonate with the public, until we try them out. I've often been very surprised about which ideas really get traction, and which don't. I've been surprised that meme accounts such as @AISafetyMemes have been pretty influential. I've also been amazed at how (unwittingly) effective Yann LeCun's recklessly anti-safety tweets have been at making people wary of the AI industry and its hubris.

This unpredictability of public responses might seriously limit the benefits of carefully planned, centrally organized activism about AI risk. It might be best just to encourage everybody who's interested to try out some public arguments, get feedback, pay attention to what works, identify common misunderstandings and pain points, share tactics with like-minded others, and iterate.

Also, lack of formal central organization limits many of the reputational risks of social media activism. If I say something embarrassing or stupid as my Twitter persona @primalpoly, that's just a reflection on that persona (and to some extent, me), not on any formal organization. Whereas if I was the grand high vice-invigilator (or whatever) in some AI safety group, my bad tweets could tarnish the whole safety group.

My hunch is that a fast, agile, grassroots, decentralized campaign of raising AI X risk awareness could be much more effective than the kind of carefully-constructed, clearly-missioned, reputationally-paranoid organizations that EAs have traditionally favored.

RagingAgainstDarkness

I'm convinced. Raising public awareness is the most important thing to do now. Is there a ready-made presentation I can use to communicate more effectively to the general public? A presentation that explains AGI risks from the ground up?

Greg_Colbourn ⏸️

Pauseai.info is now up. Not sure if this is exactly what you are looking for, but covers the basic arguments.

Otto

I don't know if everyone should drop everything else right now, but I do agree that raising awareness about AI xrisks should be a major cause area. That's why I quit my work on the energy transition about two years ago to found the Existential Risk Observatory, and this is what we've been doing since (resulting in about ten articles in leading Dutch newspapers, this one in TIME, perhaps the first comms research, a sold out debate, and a passed parliamentary motion in the Netherlands).

I miss two significant things on the list of what people can do to help:

1) Please, technical people, work on AI Pause regulation proposals! There is basically one paper now, possibly because everyone else thought a pause was too far outside the Overton window. Now we're discussing a pause anyway and I personally think it might be implemented at some point, but we don't have proper AI Pause regulation proposals, which is a really bad situation. Researchers (both policy and technical), please fix that, fix it publicly, and fix it soon!
2) You can start institutes or projects that aim to inform the societal debate about AI existential risk. We've done that and I would say it worked pretty well so far. Others could do the same thing. Funders should be able to choose from a range of AI xrisk communication projects to spend their money most effectively. This is currently really not the case.

Greg_Colbourn ⏸️

Great work you are doing with Existential Risk Observatory, Otto! Fully agree with your points too - have added them to the post. The Shavit paper is great btw, and the ideas should be further developed as a matter of priority (we need to have working mechanisms ready to implement).

Otto

Yes exactly!

Darren McKee

FYI, I'm working on a book about the risks of AGI/ASI for a general and I hope to get it out within 6 months. It likely won't be as alarmist as your post but will try to communicate the key messages, the importance, the risks, and the urgency. Happy to have more help.

Greg_Colbourn ⏸️

Cool, but a lot is likely to happen in the next 6 months! Maybe consider putting it online and updating it as you go? I feel like this post I wrote is already in need of updating (with mention of the H100 "summoning portals" already in the pipeline, CthuluGPT, Stability.ai, discussion at Zuzalu last week, Pause AI.)

Darren McKee

I'm definitely aware of that complication but I don't think that is the best way to broader impact. Uncertainty abounds. If I can get it out in 3 months, I will.

yefreitor

Common objections to this narrative are that there won’t be enough compute, or data, for this to happen. These don’t hold water after a cursory examination of our situation. We are nowhere near close to the physical limits to computation (the theoretical limit set by physics is, to a first approximation, as many orders of magnitude above current infrastructure as current infrastructure is above the babbage engine).

And we are nowhere, nowhere, nowhere, nowhere near being able to even begin approaching them. We're closer to nanotech. We're closer to dyson spheres. I can't emphasize this enough. It's like an iron-age armorsmith worrying about the physical limits of tensile strength.

Hardware overhang is a legitimate concern but you hurt your argument quite badly by mixing it with this stuff. In the whole history of the universe no one will ever be hit in the head with an entire ringworld. It's steel you have to worry about.

Greg_Colbourn ⏸️

Noted. It's the sentence after that (below) that is the more important one, so perhaps that should go first (and the rest as a footnote [Edit: I've now made these changes to OP]).

We're on ~10^18 FLOPS for large clusters at the moment^[14], but there are likely enough GPUs available for 100 times this^[15]. And the cost of this – ~$10B - is affordable for many large tech companies and national governments, and even individuals!

Harlan

Great post!

"All that is stopping them being even more powerful is spending on compute. Google & Microsoft are worth $1-2T each, and $10B can buy ~100x the compute used for GPT-4. Think about this: it means we are already well into hardware overhang territory^[5]."

I broadly agree with the point that compute could be scaled up significantly, but I want to add a few notes about the claim that $10B buys 100x the compute of GPT-4.

Altman said "more" when asked if GPT-4 had cost $100M to train. We don't know how much more. But PaLM seems to have only cost $9M-$23M so $100M is probably reasonable.

If OpenAI was buying up 100x the compute of GPT-4, maybe that would be a big enough spike in demand for GPUs that they would become more expensive. I'm pretty uncertain about what to expect there, but I estimated that PaLM used the equivalent of 0.01% of the world's current GPU/TPU computing capacity for 2 months. GPT-4 seems to be bigger than PaLM, so 100x the compute used for it might be the equivalent of more than 1% of the world's existing GPU/TPU computing capacity for 2 months.

Greg_Colbourn ⏸️

Thanks! Interested in getting more stats on global GPU/TPU capacity (there seems to be a dearth of good stats available). In your AI Impacts report, you mention paywalled reports. How much are they to access? (DM me and I can pay).

Epoch AI estimate that GPT-4 costs only $40M (not sure if that is today's price, with cost improvements since last year?)

100x the compute used for it might be the equivalent of more than 1% of the world's existing GPU/TPU computing capacity for 2 months.

1% seems well within reach for the biggest players :(

Greg_Colbourn ⏸️

I submitted this to the OpenPhil AI Worldviews Contest on 31st May with a few additions and edits - this pdf version is most up to date.

Greg_Colbourn ⏸️

Matthew Barnett's compute-based framework for thinking about the future of AI corroborates my view that data is not likely to be a bottleneck. Also, contrary to the section "against very short timelines", I argue that in fact, the data/framework used is enough to make one even more worried than I am in the OP. 1 OOM FLOP more than I previously said ("100x the compute used for GPT-4") is likely available basically now to some actors; or 4 OOM including algorithmic improvements that come "for free" with compute scaling! This (10^28-10^31 FLOP) means AGI is possible this year.

Comments

More from the author

Pause House, Blackpool

Greg_Colbourn ⏸️·9mo ago·1m read

Funding circle aimed at slowing down AI - looking for participants

Greg_Colbourn ⏸️·2y ago·2m read

Timelines are short, p(doom) is high: a global stop to frontier AI development until x-safety consensus is our only reasonable hope

Greg_Colbourn ⏸️·2y ago·10m read

Curated and popular this week

What would an animal-aligned AI be aligned to?

Aidan Kankyoku, Anima International·1w ago·Curated 23h ago·15m read

This is a crosspost from the new Animal Welfare Alignment Newsletter by Anima International. You can subscribe on Substack if you are interested in following these efforts. Audio reading also available on Substack. The goals of this post are to: 1. Raise a question I see as crucially important to the goal of aligning AI to animal welfare...

158

The first video from Giving What We Can's new channel is out now!

JustinPortela·2d ago·1m read

Hello! I'm Justin Portela. I got hired by GWWC to make YouTube videos after AI in Context did such a kickass job. My channel is using that same cinematic, high-production value beauty to talk about everything in the EA universe that isn't AI. ...

144

Maybe do the thing you wish CEA would do

alejoacelas 🔸·1w ago·2m read

I used AI to fix transcription errors, rerrarange the ideas, and suggest tweaks to the title and some sentences. Three of the most exciting projects to come out of EA in recent years are, in a vague sense, CEA spinouts: * Kairos is directly a spinout of CEA and now handles most support for university AI safety groups. Basically everyone I've found who knows them is really excited about what they do * NEST is an opinionated ideas-fi...

Recent opportunities to take action

EA Switzerland is Hiring: 🇨🇭Impact Cohort Manager

Erik Jentzen·5h ago·4m read

Find funding, fast

Austin·1d ago·3m read

173

Possible mistake EAs are making and shout out to Pause AI UK

Michelle_Hutchinson·2w ago·4m read

^{^}

Although it looks like they are highly specialised on viral threats. They would need totally new teams to be formed for AI. Maybe Hinton should chair?

^{^}

seems highly unlikely, but please say if you think there are promising solutions here

Peter Slattery 🔸

What do we need for the AI Safety movement to properly do and benefit from outreach work?

I think that doing effective collective outreach will require us to be more centralised and coordinated.

Here are my related (low confidence) intuitions (based on this and related conversations mainly) for what to do next:

It's a big ask, but it might be our best shot.

Why hasn't somebody done this already?

As I see it, the main reason for all of the above is a lack of shared language and understanding which merged because of how the AI safety community developed.

Perhaps because of this no-one who could curate preferences and set a direction (e.g., Will/Holden/Paul) feels confident to do so.

What happens if we keep going as we are?

Various other more centralised groups with leaders like Sam Altman, Tristan Harris, Tina Gebru etc will drive the conversations and changes. That might be for the best, but I suspect not.

These people could very well end up in isolated communities (e.g., just like many vegan activists I see who are mainly just reaching vegan followers on social media).

They could poison the well and make people associate AI safety with poorly informed and overconfident pessimists.

If people engage in civil disobedience we could end being feared and hated and subsequently excluded from consideration and conversation.

Our actions could create abiding associations that will damage later attempts to persuade by more persuasive sources.

This could be the unilateralist's curse brought to life.

Other thoughts/suggestions

Test the communication in small scale (e.g., with a small sample of people on mechanical turk or with friends) before you do large scale outreach

Don’t do civil disobedience unless you have read the literature about when and where it works (and maybe just don’t do it - that could backfire badly).

Think about the AI Safety ecosystem and indirect ways to get more of what you want by influencing/aiding people or processes within it:

For instance, I'd like for progress on questions like:

- When we have evidence for a strategy/action, then: Who needs to do what differently? Who do we need to communicate to, and what do we know is persuasive to them/how can we test that?

- Who could work with others for mutual benefit but doesn’t realise it yet?

I am thinking about, and doing, a little of some of these things, but have other obligations for 3-6 months and some uncertainty about whether I am well suited to do them.

^{^}

Please note here that I’m not a fan of the “Death With Dignity” vibe. I hope the more pragmatically motivated and hopeful vibe I’m trying to present here comes across in this post (at least at the end).

^{^}

If your first reaction to this is that it’s obviously wrong, I invite you to reply detailing your reasoning. See also the footnote 4 on the accompanying P(doom|AGI) is high post, that mentions a bounty of $1000.

^{^}

Note there is already the precedent of GATO.

^{^}

Perhaps if they were actually called this, rather than misnomered as Large Language Models, then people would be taking this more seriously!

^{^}

Microsoft’s R&D spend was $25B last year. And the US government can afford to train a model 1000x larger than GPT-4. (And no, we probably don’t need to be worried about China).

^{^}

There is the caveat in the referenced paper of "Nevertheless, the fact that we cannot rule out [data] contamination represents a significant caveat to our findings." - never have I more hoped for something to be true!

^{^}

We have already blasted past the conventional wisdom of:
“☐ Don’t teach it to code: this facilitates recursive self-improvement
☐ Don’t connect it to the internet: let it learn only the minimum needed to help us, not how to manipulate us or gain power
☐ Don’t give it a public API: prevent nefarious actors from using it within their code
☐ Don’t start an arms race: this incentivizes everyone to prioritize development speed over safety.”
“Industry has collectively proven itself incapable to self-regulate, by violating all of these rules.” (Tegmark, 2023).

^{^}

See next section: The default outcome of AGI is doom.

^{^}

You may be thinking: “no; what we need is a ‘Manhattan Project’ on Alignment!” Well, we need that as well as a Pause, not in lieu of one. It’s too late for alignment to be solved in time now, even with the Avengers working on it. This was last year’s best plan. And I tried (to persuade the people who could do it, to do it. But failed.) This year’s best plan is to prioritise the Pause.

^{^}

NeXt is also the name of an AI in the 2020 TV series Next, which depicts a takeover scenario involving an “Alexa” type AI that becomes superhumanly intelligent (well worth a watch imo).

^{^}

The generation after next of foundation model AI. Following generations are labelled NextAI3, NextAI4 etc.

^{^}

OpenAI and Google’s opsec is nowhere near good enough to prevent this.

^{^}

It’s unlikely that the runway AI that we’re on a path to create will be sentient. There is some speculation that recurrent neural networks of sufficient complexity could be, but that is far from established, and is not the trajectory we are on.

^{^}

4-7 months training time ~10^7 seconds; ~10^25 FLOP used to train; -> 10^18 FLOPS.

^{^}

Nvidia reported $3.6B annualised in AI chip sales in Q4 2022, and Nvidia CEO Jensen Huang estimated in Feb 2023 that “Now you could build something like a large language model, like a GPT, for something like $10, $20 million”.

^{^}

If you don't think this is enough to pose a danger, Moore's Law and associated other trends are still in effect (despite some evidence for a slowdown). And we are nowhere near close to the physical limits to computation (the theoretical limit set by physics is, to a first approximation, as many orders of magnitude above current infrastructure as current infrastructure is above the babbage engine).

^{^}

3B android phones (how many people would bother to opt out if it was part of the standard Terms & Conditions?), recording 3,300 words/day (a conservative estimate) each, would be 10 trillion words.

^{^}

It’s even possible they are doing this already, given the precedent OpenAI has set around disregarding data privacy. This would be in keeping with the big tech attitude of breaking the rules and taking the financial/legal hit after (that is usually much less than the profits gained from the transgression). See also: the NSA/Five Eyes.

^{^}

100 days of recording at the rate calculated in the above footnote would be 1 quadrillion, ~1000x what GPT-4 was trained on.

^{^}

Accessing these may not be as easy as with phone mics, but that doesn’t mean that a determined AGI+human group couldn’t hack their way to significant levels of data harvesting from them via fear or favour.

^{^}

Placed in inverted commas because it’s better to think of it as an unconscious digital alien (see next point).

^{^}

In Terminator 2, Skynet becomes “self-aware”; I think using the term “situationally aware” instead would’ve added more realism. Maybe it’s time for a remake of T2. (Then again, is there time enough left for that?)

^{^}

Re doom “only” being 50%: "it's not suicide, it's a coin flip; heads utopia, tails you're doomed" seems like quibbling at this point. [Regarding the word you're in "you're doomed" in the above - I used that instead of "we're doomed", because when CEOs hear "we're", they're probably often thinking "not we're, you're. I'll be alright in my secure compound in NZ". But they really won't! Do they think that if the shit hits the fan with this and there are survivors, there won't be the vast majority of the survivors wanting justice?]

^{^}

This is my approximate estimate for P(doom|AGI). And I’ll note here that the remaining 10% for “we’re fine” is nearly all exotic exogenous factors (related to the simulation hypothesis, moral realism being true - valence realism?, consciousness, DMT aliens being real etc), that I really don't think we can rely on to save us!

^{^}

You might be thinking, “but they are trapped in an inadequate equilibrium of racing”. Yes, but remember it’s a suicide race! At some point someone really high up in an AI company needs to turn their steering wheel. They say they are worried, but actions speak louder than words. Take the financial/legal/reputational hit and just quit! Make a big public show of it. Pull us back from the brink.
Update: 3 days after I wrote the above, Geoffrey Hinton, the “Godfather of AI”, has left Google to warn of the dangers of AI.

^{^}

As an intuition pump, imagine a million John von Neumann-level smart alien hackers each operating at a million times faster than a human, let loose on the internet, and the chaos that that would cause to global infrastructure.

^{^}

And remember that AI is already leading to catastrophic outcomes for individuals (suicide, fraud).

^{^}

The demo in the linked video looks awfully like a precursor to Skynet…

^{^}

And there is also potential for astronomical levels of future suffering.

^{^}

There was the case of the Manhattan Project scientists worrying about the first nuclear bomb test becoming a self-sustaining chain reaction and burning off the whole of our planet's atmosphere. But in that case, the richest companies in the world weren’t racing toward exploding a nuke before the scientists could get the calculations done to prove the tech was safe from this failure mode.

^{^}

I’m not sure why my signature still hasn’t been processed, but I signed it on the first day!

^{^}

It should’ve been more obvious that this was the way to go, earlier.

^{^}

Although maybe this is changing a bit too.

^{^}

Via financial incentive and social conditioning.

^{^}

And whilst the leaders of big AI companies (OpenAI, Deepmind, Anthropic) have expressed the desire to share the massive future wealth gained from AI in a globally equitable manner. The details of how such a Windfall will be managed have yet to be worked out.

^{^}

We press our fire alarms in the UK and Europe (and Australia/NZ). They are being pulled in the US too though! Are they being pressed/pulled on other continents too? Please comment if you know of examples!

^{^}

The linked post was ahead of its time. The fact that it was redacted is sad to see now (especially the crossed out parts).

^{^}

It’s unreasonable to want empirical evidence for existential risk; you can’t collect data when you’re dead!

^{^}

So not RLHF! Whether sufficient alignment has been achieved needs to be verified and agreed upon by multiple committees of independent specialists (such as ARC).

^{^}

Including from the most populous countries, most of which have not had any meaningful say over the development of the AI technology that is likely to destroy them (e.g. India, Indonesia, Pakistan, Nigeria, Brazil, Bangladesh, Mexico).

^{^}

This makes me think of the film Interstellar. Not sure if there is a useful metaphor to draw from it, but the film depicts an intense struggle for human survival; maybe it’s memeable material (this scene is a classic).

^{^}

The linked article, controversially titled We Must Declare Jihad Against A.I. – although note that Jihad in this case is referring to Bulterian Jihad, the concept from the fictional Dune universe - a war on “thinking machines” – contains the quotes: “Policymakers should consider going several steps further than the moratorium idea proposed in the recent open letter, toward a broader indefinite AI ban”, “The prevention of an AGI would be an explicit objective of this regulatory regime.”, and “a basic rule may be: The more generalized, generative, open-ended, and human-like the AI, the less permissible it ought to be—a distinction very much in the spirit of the Butlerian Jihad, which might take the form of a peaceful, pre-emptive reformation, rather than a violent uprising.”

^{^}

Yes, even China imprisons people for this.

^{^}

It’s funny to think back just 6 months, and how the FTX crisis seemed all enveloping, an existential crisis for EA. This seems positively quaint now. Oh to go back to those days of mere ordinary crisis.

^{^}

Only this is at least 100x worse.

^{^}

AI Risk is like Terminator; Stop Saying it's Not

^{^}

I’m joking and using memes here as a coping mechanism: I’m actually really fucking scared.

^{^}

Being proactive about this has certainly helped my mental health, but ymmv.

^{^}

The first few suggestions may seem a little trite in light of all the above. But you’ve got to start somewhere. And I know that EAs can be good at scaling projects up rapidly when they are driven to do so.

^{^}

I separate out Twitter here as it is arguably the biggest driver of global opinion.

^{^}

You may think that this isn’t important, but memes are effective drivers of culture.

^{^}

You can reply to OpenAI and Google Deepmind executives’ tweets with this one.

^{^}

Someone should also create and populate "Global AGI Moratorium" tags on the EA Forum and LessWrong.

^{^}

For those not in the Highly Speculative EA Capital Accumulation Facebook group, the post reads: “There's no point in trying to become a billionaire by 2030 if the world ends before then.

I'm concerned enough about AGI x-risk now, in this new post-GPT-4+plugins+AutoGPT era, that I intend to liquidate a significant fraction of my net worth over the next year or two, in a desperate attempt to help get a global moratorium on AGI (/Pause) in place. Without this breathing space to allow for alignment to catch up, I think we're basically done for in the next 2-5 years.

Have been posting a lot on Twitter. For the next year or two I'm an "AI Notkilleveryoneist" above being an EA. I hope we can get back to EA business as usual soon (and I can relax and enjoy my retirement a bit), but this is an unprecedented global emergency that needs resolving. Is anyone else here thinking the same way?”

^{^}

Shavit's paper is great and the ideas in it should be further developed as a matter of priority.

^{^}

I wrote this post in ~2 days (although I developed a lot of the points on Twitter over the preceding couple of weeks) and had one round of feedback over ~2 days.

^{^}

Sam Altman is being mean to Yudkowsky here, but maybe he has a point. OpenAI and Anthropic were both founded over concern for AI Alignment, but have now, I think, become two of the main drivers of AGI x-risk.

^{^}

If your bottleneck for doing this is money, talk to me.

^{^}

Maybe I can even get to enjoy my retirement a bit. One can hope.

^{^}

To be fair to EA and x-risk, it’s actually more like life saving medical treatments on Malaysia Airlines Flight 370 (as per my meme above).

^{^}

Or (re the above footnoted reference), running to the cockpit, grabbing the controls, and steering away from the ocean abyss..

AGI rising: why we are in a new era of acute risk and increasing public awareness, and what to do now

AGI rising: why we are in a new era of acute risk and increasing public awareness, and what to do now

We are in a new era of acute risk from AGI

The default outcome of AGI is doom

Increasing public awareness

What we need to happen to get out of the acute risk period

What you can and perhaps should do now