Today, we’re announcing that Amazon will invest up to $4 billion in Anthropic. The agreement is part of a broader collaboration to develop reliable and high-performing foundation models.


(Thread continues from there with more details -- seems like a notable major development!)

38

1
0

Reactions

1
0
Comments34


Sorted by Click to highlight new comments since:

If this is true, I will update even further in the direction of the creation of anthropic being a net negative to the world.

Amazon is a massive multinational driven by profit almost alone, that will be continuously pushing for more and more, while paying less and less attention to safety.

It surprised me a bit that anthropic would allow this to happen.

Disagree. The natural, no-Anthropic, counterfactual is one in which Amazon invests billions into an alignment-agnostic AI company. On this view, Anthropic is levying a tax on AI-interest where the tax pays for alignment. I'd put this tax at 50% (rough order of magnitude number).

If Anthropic were solely funded by EA money, and didn't capture unaligned tech funds this would be worse. Potentially far worse since Anthropic impact would have to be measured against the best alternative altruistic use of the money.

I suppose you see this Amazon investment as evidence that Anthropic is profit motivated, or likely to become so. This is possible, but you'd need to explain what further factors outweigh the above. My vague impression is that outside investment rarely accidentally costs existing stakeholders control of privately held companies. Is there evidence on this point?

I think the modal no-Anthropic counterfactual does not have an alignment-agnostic AI company that's remotely competitive with OpenAI, which  means there's no external target for this Amazon investment.  It's not an accident that Anthropic was founded by former OpenAI staff who were substantially responsible for OpenAI's earlier GPT scaling successes.

What do you think the bottleneck for this alternate AI company’s competitiveness would be? If it’s talent, why is it insurmountable? E.g. what would prevent them from hiring away people from the current top labs?

There are alternatives - x.AI and Inflection. Arguably they only got going because the race was pushed to fever pitch by Anthropic splitting from OpenAI.

It seems more likely to me that they would have gotten started anyway once ChatGPT came out. Although I was interpreting the counterfactual as being if Anthropic had declined to partner with Amazon, rather than if Anthropic had not existed.

I'm not sure if they would've ramped up quite so quick (i.e. getting massive investment) if it wasn't for the race heating up with Anthropic entering. Either way, it's all bad, and a case of which is worse.

This is assuming that Anthropic is net positive even in isolation. They may be doing some alignment research, but they are also pushing the capabilities frontier. They are either corrupted by money and power, or hubristically think that they can actually save the world following their strategy, rather than just end it. Regardless, they are happy to gamble hundreds of millions of lives (in expectation) without any democratic mandate. Their "responsible scaling" policy is anything but (it's basically an oxymoron at this stage, when AGI is on the horizon and alignment is so far from being solved).

Yeah, not sure how much this is good news and the level of interference and vested interests that will inevitably come up. 

I am curious if the FTX stake in Anthropic is now valuable enough to plausibly bail out FTX? Or at least put a dent in the amount owed to customers who were scammed?

I've lost track of the gap between assets and liabilities at FTX, but this is a $4B investment for a minority stake, according to news reports. Which implies Anthropic has a post-money valuation of at least $8B. Anthropic was worth $4.6B in June according to this article. So the $500M stake reportedly held by FTX should might be worth around double whatever it was worth in June, and possibly quite a bit more.

Edit: this article suggests the FTX asset/liability gap was about $2B as of June. So the rise in valuation of the Anthropic stake is certainly a decent fraction of that, though I'd be surprised if it's now valuable enough to cover the entire gap.

Edit 2: the math is not quite as simple as I made it seem above, and I've struck out the word "should" to reflect that. Anyway, I think the question is still the size of the minority share that Amazon bought (which has not been made public AFAICT) as that should determine Anthropic's market cap.

I do not understand Dario's[1] thought process or strategy really

At a (very rough) guess, he thinks that Anthropic alone can develop AGI safely, and they need money to keep up with OpenAI/Meta/any other competitors because they're going to cause massive harm to the world and can't be trusted to do so?

If that's true then I want someone to hold his feet to the fire on that, in the style of Gary Marcus telling the Senate hearing that Sam Altman had dodged their question on what his 'worst fear' was - make him say it in an open, political hearing as a matter of record.

  1. ^

    Dario Amodei, Founder/CEO of Anthropic

See Dario's Senate testimony from two months ago:

With the fast pace of progress in mind, we can think of AI risks as falling into three buckets:

●  Short-term risks are those present in current AI systems or that imminently will be present. This includes concerns like privacy, copyright issues, bias and fairness in the model’s outputs, factual accuracy, and the potential to generate misinformation or propaganda.

●  Medium-term risks are those we will face in two to three years. In that time period, Anthropic’s projections suggest that AI systems may become much better at science and engineering, to the point where they could be misused to cause large-scale destruction, particularly in the domain of biology. This rapid growth in science and engineering skills could also change the balance of power between nations.

●  Long-term risks relate to where AI is ultimately going. At present, most AI systems are passive and merely converse with users, but as AI systems gain more and more autonomy and ability to directly manipulate the external world, we may face increasing challenges in controlling them. There is a spectrum of problems we could face related to this, at the extreme end of which is concerns about whether a sufficiently powerful AI, without appropriate safeguards, could be a threat to humanity as a whole – referred to as existential risk. Left unchecked, highly autonomous, intelligent systems could also be misused or simply make catastrophic mistakes.

Note that there are some concerns, like AI’s effects on employment, that don’t fit neatly in one bucket and probably take on a different form in each time period.

Short-term risks are in the news every day and are certainly important. I expect we’ll have many opportunities to discuss these in this hearing, and much of Anthropic’s research applies immediately to those risks: our constitutional AI principles include attempts to reduce bias, increase factual accuracy, and show respect for privacy, copyright, and child safety. Our red-teaming is designed to reduce a wide range of these risks, and we have also published papers on using AI systems to correct their own biases and mistakes. There are a number of proposals already being considered by the Congress relating to these risks.

The long-term risks might sound like science fiction, but I believe they are at least potentially real. Along with the CEOs of other major AI companies and a number of prominent AI academics (including my co-witnesses Professors Russell and Bengio) I have signed a statement emphasizing that these risks are a challenge humanity should not neglect. Anthropic has developed evaluations designed to measure precursors of these risks and submitted its models to independent evaluators. And our work on interpretability is also designed to someday help with long-term risks. However, the abstract and distant nature of long-term risks makes them hard to approach from a policy perspective: our view is that it may be best to approach them indirectly by addressing more imminent risks that serve as practice for them.

The medium-term risks are where I would most like to draw the subcommittee’s attention. Simply put, a straightforward extrapolation of the pace of progress suggests that, in 2-3 years, AI systems may facilitate extraordinary insights in broad swaths of many science and engineering disciplines. This will cause a revolution in technology and scientific discovery, but also greatly widen the set of people who can wreak havoc. In particular, I am concerned that AI systems could be misused on a grand scale in the domains of cybersecurity, nuclear technology, chemistry, and especially biology.

Thanks for linking Dario's testimony. I actually found this extract which was closer to answering my question:

I wanted to answer one obvious question up front: if I truly believe that AI’s risks
are so severe, why even develop the technology at all? To this I have three answers: 

First, if we can mitigate the risks of AI, its benefits will be truly profound. In the next few years it could greatly accelerate treatments for diseases such as cancer, lower the cost of energy, revolutionize education, improve efficiency throughout government, and much more. 

Second, relinquishing this technology in the United States would simply hand over its power, risks, and moral dilemmas to adversaries who do not share our values. 

Finally, a consistent theme of our research has been that the best mitigations to the risks of powerful AI often also involve powerful AI. In other words, the danger and the solution to the danger are often coupled. Being at the frontier thus puts us in a strong position to develop safety techniques (like those I’ve mentioned above), and also to see ahead and warn about risks, as I’m doing today.

I know this statement would have been massively pre-prepared for the hearing, but I don't feel super convinced by it:

On his point 1) such benefits have to be weighed up against the harms, both existential and not. But just as many parts of the xRisk story are speculative, so are many of the purported benefits from AI research. I guess Dario is saying 'it could' and not it will, but for me if you want to "improve efficiency throughout government" you'll need political solutions, not technical ones.

Point 2) is the 'but China' response to AI Safety. I'm not an expert in US foreign policy strategy (funny how everyone is these days), but I'd note this response only works if you view the path to increasing capability as straightforward. It also doesn't work, in my mind, if you think there's a high chance of xRisk. Just because someone else might ignite the atmosphere, doesn't mean you should too. I'd also note that Dario doesn't sound nearly as confident making this statement as he did talking to it with Dwarkesh recently.

Point 3) makes sense if you think the value of the benefits massively outweighs the harms, so that you solve the harms as you reap the benefits. But if those harms outweigh the benefits, or you incure a substantial "risk of ruin", then being at the frontier and expanding it further unilaterally makes less sense to me.

I guess I'd want the CEOs and those with power in these companies to actually be put under the scrutiny in the political sphere which they deserve. These are important and consequential issues we're talking about, and I just get the vibe that the 'kid gloves' need to come off a bit in turns of oversight and scrutiny/scepticism.

Yeah, I think the real reason is we think we're safer than OpenAI (and possibly some wanting-power but that mostly doesn't explain their behavior).

Yes, it's total hubris. And many at OpenAI and DeepMind feel the same way - their company alone can save the world (and prevent the other AI companies from ending it). This is the other AI arms race. It's as dangerous as the e/acc one. No regard for the fact that the public does not want them to do this.

I'm basically making the same point as the parent comment, although perhaps a bit more starkly, and with the additional point about lack of democratic mandate. Yet that's on +36 karma and mine is on -6. This is why we need a separate "outside game" movement on AI x-safety.

I haven't thought about this a lot, but I don't see big tech companies working with existing frontier AI players as necessarily a bad thing for race dynamics (compared to the counterfactual). It seems better than them funding or poaching talent to create a viable competitor that may not care as much about risk - I'd guess the question is how likely we'd expect them to be successful in doing so (given that Amazon is not exactly at the frontier now)?

From what I understand, Amazon does not get a board seat for this investment. Figured that should be highlighted. Seems like Amazon just gets to use Anthropic’s models and maybe make back their investment later on. Am I understanding this correctly? 

As part of the investment, Amazon will take a minority stake in Anthropic. Our corporate governance structure remains unchanged, with the Long Term Benefit Trust continuing to guide Anthropic in accordance with our Responsible Scaling Policy. As outlined in this policy, we will conduct pre-deployment tests of new models to help us manage the risks of increasingly capable AI systems.

I hope this is just cash and not a strategic partnership, because if it is, then it would mean there is now a third major company in the AGI race.

It seems pretty clear that Amazon's intent is to have state of the art AI backing Alexa. That alone would not be particularly concerning. The problem would be if Amazon has some leverage to force Anthropic to accelerate capabilities research and neglect safety - which is certainly possible, but it seems like Anthropic wants to avoid it by keeping Amazon as a minority investor and maintaining the existing governance structure.

Judging by the example of Microsoft owning a minority stake in OpenAI (and the subsequent rush to release Bing's Sydney/GPT-4), that's not exactly comforting.

I interpret it as broadly the latter based on the further statements in the Twitter thread, though I could well be wrong.

Um, conditional on any AI labs being in a race in what way are Anthropic not already racing?

Anthropic is small compared with Google and OpenAI+Microsoft.

I would, however, not downplay their talent density.

Ah, I thought you were implying that Anthropic weren't already racing when you were actually pointing at Amazon (a major company) joining the race. I agree that Anthropic is not a "major" company.

It seems pretty overdetermined to me that Amazon and Apple will join either join the race by acquiring a company or by reconfiguring teams/hiring. I'm a bit confused about whether I want it to happen now, or later. I'd guess later.

Curated and popular this week
 ·  · 12m read
 · 
Economic growth is a unique field, because it is relevant to both the global development side of EA and the AI side of EA. Global development policy can be informed by models that offer helpful diagnostics into the drivers of growth, while growth models can also inform us about how AI progress will affect society. My friend asked me to create a growth theory reading list for an average EA who is interested in applying growth theory to EA concerns. This is my list. (It's shorter and more balanced between AI/GHD than this list) I hope it helps anyone who wants to dig into growth questions themselves. These papers require a fair amount of mathematical maturity. If you don't feel confident about your math, I encourage you to start with Jones 2016 to get a really strong grounding in the facts of growth, with some explanations in words for how growth economists think about fitting them into theories. Basics of growth These two papers cover the foundations of growth theory. They aren't strictly essential for understanding the other papers, but they're helpful and likely where you should start if you have no background in growth. Jones 2016 Sociologically, growth theory is all about finding facts that beg to be explained. For half a century, growth theory was almost singularly oriented around explaining the "Kaldor facts" of growth. These facts organize what theories are entertained, even though they cannot actually validate a theory – after all, a totally incorrect theory could arrive at the right answer by chance. In this way, growth theorists are engaged in detective work; they try to piece together the stories that make sense given the facts, making leaps when they have to. This places the facts of growth squarely in the center of theorizing, and Jones 2016 is the most comprehensive treatment of those facts, with accessible descriptions of how growth models try to represent those facts. You will notice that I recommend more than a few papers by Chad Jones in this
LintzA
 ·  · 15m read
 · 
Introduction Several developments over the past few months should cause you to re-evaluate what you are doing. These include: 1. Updates toward short timelines 2. The Trump presidency 3. The o1 (inference-time compute scaling) paradigm 4. Deepseek 5. Stargate/AI datacenter spending 6. Increased internal deployment 7. Absence of AI x-risk/safety considerations in mainstream AI discourse Taken together, these are enough to render many existing AI governance strategies obsolete (and probably some technical safety strategies too). There's a good chance we're entering crunch time and that should absolutely affect your theory of change and what you plan to work on. In this piece I try to give a quick summary of these developments and think through the broader implications these have for AI safety. At the end of the piece I give some quick initial thoughts on how these developments affect what safety-concerned folks should be prioritizing. These are early days and I expect many of my takes will shift, look forward to discussing in the comments!  Implications of recent developments Updates toward short timelines There’s general agreement that timelines are likely to be far shorter than most expected. Both Sam Altman and Dario Amodei have recently said they expect AGI within the next 3 years. Anecdotally, nearly everyone I know or have heard of who was expecting longer timelines has updated significantly toward short timelines (<5 years). E.g. Ajeya’s median estimate is that 99% of fully-remote jobs will be automatable in roughly 6-8 years, 5+ years earlier than her 2023 estimate. On a quick look, prediction markets seem to have shifted to short timelines (e.g. Metaculus[1] & Manifold appear to have roughly 2030 median timelines to AGI, though haven’t moved dramatically in recent months). We’ve consistently seen performance on benchmarks far exceed what most predicted. Most recently, Epoch was surprised to see OpenAI’s o3 model achieve 25% on its Frontier Math
Omnizoid
 ·  · 5m read
 · 
Edit 1/29: Funding is back, baby!  Crossposted from my blog.   (This could end up being the most important thing I’ve ever written. Please like and restack it—if you have a big blog, please write about it). A mother holds her sick baby to her chest. She knows he doesn’t have long to live. She hears him coughing—those body-wracking coughs—that expel mucus and phlegm, leaving him desperately gasping for air. He is just a few months old. And yet that’s how old he will be when he dies. The aforementioned scene is likely to become increasingly common in the coming years. Fortunately, there is still hope. Trump recently signed an executive order shutting off almost all foreign aid. Most terrifyingly, this included shutting off the PEPFAR program—the single most successful foreign aid program in my lifetime. PEPFAR provides treatment and prevention of HIV and AIDS—it has saved about 25 million people since its implementation in 2001, despite only taking less than 0.1% of the federal budget. Every single day that it is operative, PEPFAR supports: > * More than 222,000 people on treatment in the program collecting ARVs to stay healthy; > * More than 224,000 HIV tests, newly diagnosing 4,374 people with HIV – 10% of whom are pregnant women attending antenatal clinic visits; > * Services for 17,695 orphans and vulnerable children impacted by HIV; > * 7,163 cervical cancer screenings, newly diagnosing 363 women with cervical cancer or pre-cancerous lesions, and treating 324 women with positive cervical cancer results; > * Care and support for 3,618 women experiencing gender-based violence, including 779 women who experienced sexual violence. The most important thing PEPFAR does is provide life-saving anti-retroviral treatments to millions of victims of HIV. More than 20 million people living with HIV globally depend on daily anti-retrovirals, including over half a million children. These children, facing a deadly illness in desperately poor countries, are now going