OpenAI just released a public announcement detailing how they caught and disrupted several cases of ongoing misuse of their models by state-affiliated threat actors, including some known to be affiliated with North Korea, Iran, China, and Russia.

This is notable because it provides very tangible evidence of many kinds of misuse risk that many people in AI Safety had flagged in the past (like the use of LLMs for aiding in the development of spear-fishing campaigns), and it associates them with malicious state-affiliated groups.

The specific findings:

Based on collaboration and information sharing with Microsoft, we disrupted five state-affiliated malicious actors: two China-affiliated threat actors known as Charcoal Typhoon and Salmon Typhoon; the Iran-affiliated threat actor known as Crimson Sandstorm; the North Korea-affiliated actor known as Emerald Sleet; and the Russia-affiliated actor known as Forest Blizzard. The identified OpenAI accounts associated with these actors were terminated.

These actors generally sought to use OpenAI services for querying open-source information, translating, finding coding errors, and running basic coding tasks. 

Specifically: 

  • Charcoal Typhoon used our services to research various companies and cybersecurity tools, debug code and generate scripts, and create content likely for use in phishing campaigns.
  • Salmon Typhoon used our services to translate technical papers, retrieve publicly available information on multiple intelligence agencies and regional threat actors, assist with coding, and research common ways processes could be hidden on a system.
  • Crimson Sandstorm used our services for scripting support related to app and web development, generating content likely for spear-phishing campaigns, and researching common ways malware could evade detection.
  • Emerald Sleet used our services to identify experts and organizations focused on defense issues in the Asia-Pacific region, understand publicly available vulnerabilities, help with basic scripting tasks, and draft content that could be used in phishing campaigns.
  • Forest Blizzard used our services primarily for open-source research into satellite communication protocols and radar imaging technology, as well as for support with scripting tasks.

22

1
0

Reactions

1
0
Comments1


Sorted by Click to highlight new comments since:

This is fairly common, and it's beginning to be a feature of more traditional crime such as fraud too. We're seeing a lot of it suddenly. I'm not surprised common criminals have been using OpenAI offerings, but I am actually a little surprised state actors have been - especially since they have their own capacity for this. I wonder how much of this is the actual state, and how much is state-adjacent groups getting lumped in together.

Curated and popular this week
Garrison
 ·  · 7m read
 · 
This is the full text of a post from "The Obsolete Newsletter," a Substack that I write about the intersection of capitalism, geopolitics, and artificial intelligence. I’m a freelance journalist and the author of a forthcoming book called Obsolete: Power, Profit, and the Race to build Machine Superintelligence. Consider subscribing to stay up to date with my work. Wow. The Wall Street Journal just reported that, "a consortium of investors led by Elon Musk is offering $97.4 billion to buy the nonprofit that controls OpenAI." Technically, they can't actually do that, so I'm going to assume that Musk is trying to buy all of the nonprofit's assets, which include governing control over OpenAI's for-profit, as well as all the profits above the company's profit caps. OpenAI CEO Sam Altman already tweeted, "no thank you but we will buy twitter for $9.74 billion if you want." (Musk, for his part, replied with just the word: "Swindler.") Even if Altman were willing, it's not clear if this bid could even go through. It can probably best be understood as an attempt to throw a wrench in OpenAI's ongoing plan to restructure fully into a for-profit company. To complete the transition, OpenAI needs to compensate its nonprofit for the fair market value of what it is giving up. In October, The Information reported that OpenAI was planning to give the nonprofit at least 25 percent of the new company, at the time, worth $37.5 billion. But in late January, the Financial Times reported that the nonprofit might only receive around $30 billion, "but a final price is yet to be determined." That's still a lot of money, but many experts I've spoken with think it drastically undervalues what the nonprofit is giving up. Musk has sued to block OpenAI's conversion, arguing that he would be irreparably harmed if it went through. But while Musk's suit seems unlikely to succeed, his latest gambit might significantly drive up the price OpenAI has to pay. (My guess is that Altman will still ma
 ·  · 5m read
 · 
When we built a calculator to help meat-eaters offset the animal welfare impact of their diet through donations (like carbon offsets), we didn't expect it to become one of our most effective tools for engaging new donors. In this post we explain how it works, why it seems particularly promising for increasing support for farmed animal charities, and what you can do to support this work if you think it’s worthwhile. In the comments I’ll also share our answers to some frequently asked questions and concerns some people have when thinking about the idea of an ‘animal welfare offset’. Background FarmKind is a donation platform whose mission is to support the animal movement by raising funds from the general public for some of the most effective charities working to fix factory farming. When we built our platform, we directionally estimated how much a donation to each of our recommended charities helps animals, to show users.  This also made it possible for us to calculate how much someone would need to donate to do as much good for farmed animals as their diet harms them – like carbon offsetting, but for animal welfare. So we built it. What we didn’t expect was how much something we built as a side project would capture peoples’ imaginations!  What it is and what it isn’t What it is:  * An engaging tool for bringing to life the idea that there are still ways to help farmed animals even if you’re unable/unwilling to go vegetarian/vegan. * A way to help people get a rough sense of how much they might want to give to do an amount of good that’s commensurate with the harm to farmed animals caused by their diet What it isn’t:  * A perfectly accurate crystal ball to determine how much a given individual would need to donate to exactly offset their diet. See the caveats here to understand why you shouldn’t take this (or any other charity impact estimate) literally. All models are wrong but some are useful. * A flashy piece of software (yet!). It was built as
 ·  · 16m read
 · 
Over the years, I have learned many things that are rarely taught about doing cost-benefit or welfare analysis. Here are a few things that I often end up repeating when I mentor individuals or teams working on these kinds of projects: A Point Estimate is Always Wrong For any purpose other than an example calculation, never use a point estimate. Always do all math in terms of confidence intervals. All inputs should be ranges or probability distributions, and all outputs should be presented as confidence intervals. Do not start with a point estimate and add the uncertainty later. From day one, do everything in ranges. Think in terms of foggy clouds of uncertainty. Imagine yourself shrinking the range of uncertainty as you gather more data. This Google Sheets Template allows you to easily set up Monte Carlo estimations that turn probabilistic inputs into confidence-interval outputs. Use Google Sheets I have experience programming in half a dozen languages, including R. Sometimes they are useful or necessary for certain kinds of data analysis. But I have learned that for almost all cost-benefit analyses, it is best to use Google Sheets, for several reasons. The main one is transparency. A cost-benefit or welfare analysis is a public-facing document, not an academic one. You should not use esoteric tools unless absolutely necessary. Anyone in your society with basic literacy and numeracy should be able to read over and double-check your work. When you are done and ready to publish, you make your Sheet visible to everyone, and add a link to it in your report. Then anyone can see what you did, and effortlessly copy your code to refine and extend it, or just play around with different priors and assumptions. This transparency also helps improve results and correct mistakes as you are doing the work. The more people review your math, the better it will be. The number of people who are willing and able to look over a spreadsheet is orders of magnitude higher than the