Hide table of contents

What have recent changes in the information ecosystem been?

 

Events of the last few years, including the war in Ukraine and the pandemic, have accelerated structural shifts towards more digital, mobile, and platform-dominated media environments. The widespread use of synthetic content has been observed in recent conflicts, ranging from Ukraine to Sudan to Gaza. Social media platforms have been changing rapidly, in the last year Facebook published updates related to Meta, Reels, AI-related content. Twitter under Elon Musk has loosened moderation guidelines. TikTok has become one of the fastest-growing platforms ever, reflecting the growth in short form video content. 

 

The explosive growth in the amount of content created has increased the prominence on AI led content moderation in order to scale effectively. There is also growing attention from governments on content moderation, with the EU’s Digital Services Act which, Australia’s Online Safety Act and the UK Online Safety Bill, amongst others. 

 

The interconnected nature of platforms and user networks means that disinformation is networked across many platforms, including unmoderated or encrypted spaces, making it hard to identify, track and flag or remove false content at scale across the information landscape. 

 

What does AI mean for creating mis/disinformation?

 

As the availability of LLMs increases and cost falls, this makes it easier for threat actors to create more personalised and more effective content. As content creation is becoming more automated, this reduces the financial and time costs associated with micro targeting and hyper personalization, and an improved understanding of the information environment allows threat actors to craft more compelling and effective narratives for each target segment. This also makes disinformation campaigns more difficult to detect as new content is easier to generate, preventing a need for copypasta, and the quality of deepfakes is drastically improving. 

 

What does AI mean for the spread of mis/disinformation?

 

The spread of campaigns often relies on large numbers of accounts across social media, and the perceived authenticity of the accounts is key. ML techniques allow generation of increasingly realistic profile photos, reducing the need for image scraping and the potential for reverse image searches to aid in detection of a campaign. When combined with the improvements in text generation through LLMs for bio’s and online presence, this results in en masse creation of credible accounts to spread disinformation. 

 

ML systems can also improve social engineering techniques to target influencers or so called “super-spreaders” who can organically amplify a message or campaign. Deepfakes also make it easier to impersonate experts or credible sources to amplify a message. 

 

What does AI mean to real-time information ecosystem interactions?

 

Advancements in conversational AI or chatbots could automate engament with targeted individuals. These use large volumes of data, ML, and NLP to imitate human interactions, recognizing speech and text input and generating a response. This can be used to take part in online discussions and respond to comments to stimulate controversy and disputes, and increase polarisation. 

 

As AI reduces the costs, increases the effectiveness, and reduces the ease of detection of disinformation campaigns, threat monitoring and early detection are becoming increasingly important. 

 

How does this impact global catastrophic risk?

Information ecosystem risks are both broad and dangerous, with potential impacts ranging from water security to financial stability and even amplifying/creating conflicts. Disinformation is used to influence public opinion, legitimise unpopular actions and regimes including garnering support for confrontation, alter election results, increase polarisation and extremism, and undermine the credibility of institutions, science, experts and media. 

 

Examples include the use of disinformation as a form of information warfare in the Russia Ukraine conflict, the spread of conspiracy theories around the 2016 US elections, and the fuelling of anti muslim and anti Pakistan sentiment in India. Much like climate change, this acts as a threat multiplier for catastrophic and existential risks and can significantly increase risk by polarising and destabilising the world, undermining global governance, increasing geopolitical tensions, and the risk of conflict, arms races, and more. 

 

Disinformation from generative AI has been seen in recent conflicts in Sudan, Gaza and the Ukraine with deepfakes from leaders of both sides, designed to stoke tensions and escalate conflict. The role of AI-empowered disinformation has the potential to both create and worsen conflict.

 

Disinformation is used to undermine scientific credibility and promote climate denialism and inaction. A recent report from CCDH showed that ⅓ of British children thought concerns of climate change were overblown which shows marked success with climate denialism narratives. 

 

Disinformation around nuclear weapons can be used by state or non state actors to simulate or provoke nuclear attack or response. It also increases geopolitical tensions and the likelihood of great power war. The recent election in Taiwan was supposedly subject to significant influence operations by mainland China with AI augmented disinformation.

 

Disinformation is used by threat actors and incentivised parties to downplay AI safety concerns and promote rapid AI development at the expense of safety, alignment and governance. This could significantly hamper the efforts of AI safety activities.

 

AI and disinformation: Offence vs Defence 

 

Spreading disinformation has always been more effective than combatting it. In creating and spreading disinformation, the most viral techniques and narratives can be used without regard for the truth, often preying on insecurities for the most vulnerable and tapping into visceral emotions such as fear.  The effectiveness is evident across the board, notably in the difference in click-through rates for clickbait headlines generated by ad companies Taboola and Outbrain vs generic news stories. Responding to disinformation also has a temporal disadvantage, where a counter-narrative as opposed to an initial narrative has to be established, and an anchoring effect has often occurred. 

 

Techno-fixes are also limited in their effectiveness, and certain ones such as reverse image searches place a high burden of effort for the user. Fact-checking is time consuming and laborious and an ever-decreasing proportion of information can be fact-checked as AI-generated/spread disinformation is proliferated. 

 

AI empowered tools such as Bot Sentinel, Botometer and BotSlayer classify accounts as bots based on profile features and behaviour. Tools and solutions such as Captain Fact, Claimbuster, Logically.ai and Alethea Artemis use AI to detect misinformation and disinformation at scale. However the effectiveness of detection algorithms depends on the availability of large sets of training data and quality of data labels. While detection is becoming more robust, for example within deepfake detection by looking beyond subtle signatures of particular generation tools and using underlying physical and biological signals that are hard for AI to imitate, there is a constant back and forth between AI-generated content and detection methods as both sides become more sophisticated and adapt to each other.

 

As techniques such as establishing provenance on the blockchain are established, with the use of digital watermarks, these will be likely circumvented by rogue state and non-state actors empowered by AI. The commercial incentives for creating and spreading disinformation have historically been greater than countering it, and it is difficult to see how this may change.

 

AI is likely to make disinformation much easier to create and spread more effectively, at lower human and financial cost. AI tools often place a high burden of effort for the end user so will have limited adoption and as they are developed, zero day exploits and other techniques will be developed to circumvent these tools. Although technical tools are a crucial part of the toolkit to combat mis and disinformation, they are necessary but not sufficient.

 

What is the current state of knowledge on the field?

 

Campaigners across government and civil society are often resource and time-poor and don’t know the best, evidence-informed way to respond to information threats in combatting mis/disinformation. There is a wealth of academic literature on quantitative experiments on the efficacy of interventions to combat mis/disinformation, however this is disparate, disaggregated, spans many disciplines and lacks a shared ontology. This evidence base is hard to find, navigate and interpret so tends to be ignored by the practitioner community.

 

Our Proposal: Strengthening the evidence base for societal resilience

 

This project aims to create, populate, test the effectiveness and iterate an online living database to be used to improve the effectiveness of counter disinformation campaigns and media literacy, especially around AI. This open source online living database will collate, curate and categorise empirical studies that have been run on interventions to combat mis/disinformation and extract insights from study. The insights will include characterising the relevant information threat, intervention tested, methodology details, participant characteristics and statistics from results. This will enable the comparison of different interventions for any characterised information threat. The next phase of the project will be to create an algorithm to rank interventions for any characterised information threat, based on characterised parameters such as effect size, sample size, effect duration and others. 

 

The database aims to operationalise the existing evidence base by aggregating a highly disaggregated academic field, serving both the research and practitioner community, having a large scale impact by improving the effectiveness of campaigns and media literacy initiatives run by large numbers of practitioner groups around the world. 

 

We have validated the utility of this proposition with leading academics and practitioners in the field including the University of Cambridge Social Decision-Making Lab, IRIS (LSHTM, University of Rome, University of Venice), University of Bristol, Max Planck Institute, North-Eastern University, University of Minnesota, IMT Lucca, University of Seattle, Princeton, UC Davis, the Centre for Countering Digital Hate, The European Centre of Excellence for Countering Hybrid Threats, ISD Global, CASM, Stimson Center, Public Democracy, and Climate Action Against Disinformation.

 

Learnings so far: What interventions may work? Evidence based counter campaigns & media literacy

 

While fact checking and other labelling technologies are progressing in identification of mis/disinformation online, technology development such as AI (e.g. LLM’s) makes the creation and spread of disinformation significantly quicker, easier and cheaper, and changes the tactics used, hence technofixes will constantly be catching up. Tackling mis and disinformation requires both counter specific disinformation campaigns run on particular issues but more importantly the building of individual and societal resilience through education interventions focused on media literacy, especially around emerging technologies and AI.

 

What do effective counter interventions look like?  

 

A range of counter interventions are able to counter disinformation, some regulatory, some platform-oriented but also those that can be integrated into campaigns. 

 

Debunking (providing correcting information targeted towards misconceptions of beliefs), inoculation (pre-emptive exposure to weakened forms of disinformation, and is often technique based or issue based) and adjacent messaging (providing alternative more hopeful narratives as opposed to directly refuting) are all tools that can be integrated into campaigns. 

 

AI will create a more uncertain information environment, with fatigue likely setting into citizens where fact checking tools are arduous and time consuming to use. Effective counter-interventions are highly context-specific but likely to have the following characteristics in common.

 

  1. Early detection. Information threats are much easier to respond to when nascent, when more effective techniques can be used. Deplatforming and prebunking are effective options to quash influence threats before disinformation narratives become widespread  
  2. Target knowledge acquisition. Obtaining demographic and psychographic insight into potential targets of influence operations, e.g. customers/investors help design more effective counter-campaigns
  3. Evidence-based response. As AI-related disinformation proliferates, preempting particular narratives becomes more challenging. Therefore technique based inoculation is likely to be more effective, as techniques will target weaknesses from common cognitive fallacies, which may be personalised based on psychographic elements

 

How may we want to update media literacy training in light of this? 

 

The future world is one where disinformation is more prevalent, more personalised and harder to discern. In light of this, effective media literacy to build societal resilience will need to include the following:

 

  1. How to hold information in uncertainty. People have a preference for certainty over uncertainty (certainty effect), and aiding people to develop probabilistic mindsets where information may or may not be true is pivotal
  2. How to interact with uncertain information. Communicating the uncertainty associated with information is critical to enable others to also hold information in uncertainty, and not for strength in beliefs to increase with sharing.
  3. How to recognise influence operations. Educating the public on who may be targeting them, why they do so, the techniques they use, the goals they have and how this links to particular narratives can help identify when a piece of information is more likely to be disinformation.
  4. What technical tools can be used for verifying information, reporting mis/disinformation and deplatforming

 

Who are we?

 

Say No to Disinfo is focused on improving the information ecosystem, and hence reducing existential risk by making society more robust and resilient to disruption due to mis/disinformation. We have a focus on the intersection of AI and emerging technologies with disinformation and our activities focus on improving the effectiveness of direct disinformation response, and of educational interventions to improve media literacy and societal resilience to mis/disinformation through the creation of the online living database, and the work on improving the effectiveness of media literacy for the current and future technology environment. 
 

Progress to date and our asks

 

With collaboration from leading academics and practitioners, we have completed database design, compiled an initial list of hundreds of academic papers containing thousands of experiments on counter mis/disinformation interventions, and are in the process of uploading these into the database. 

 

We are seeking to augment the academic data with field data from campaigns that have been run. If you/your organisation have any data that we could incorporate or that you could make us aware of, please let us know.

 

We are looking for volunteers to upload papers to the database, reviewing them and extracting key information. It provides a hands-on opportunity to learn from cutting edge studies whilst contributing to a living resource that will have tangible positive real world impact, used by both civil society and government. If you are interested please get in touch with us at ari@saynotodisinfo.com


 

Comments


No comments on this post yet.
Be the first to respond.
Curated and popular this week
 ·  · 10m read
 · 
I wrote this to try to explain the key thing going on with AI right now to a broader audience. Feedback welcome. Most people think of AI as a pattern-matching chatbot – good at writing emails, terrible at real thinking. They've missed something huge. In 2024, while many declared AI was reaching a plateau, it was actually entering a new paradigm: learning to reason using reinforcement learning. This approach isn’t limited by data, so could deliver beyond-human capabilities in coding and scientific reasoning within two years. Here's a simple introduction to how it works, and why it's the most important development that most people have missed. The new paradigm: reinforcement learning People sometimes say “chatGPT is just next token prediction on the internet”. But that’s never been quite true. Raw next token prediction produces outputs that are regularly crazy. GPT only became useful with the addition of what’s called “reinforcement learning from human feedback” (RLHF): 1. The model produces outputs 2. Humans rate those outputs for helpfulness 3. The model is adjusted in a way expected to get a higher rating A model that’s under RLHF hasn’t been trained only to predict next tokens, it’s been trained to produce whatever output is most helpful to human raters. Think of the initial large language model (LLM) as containing a foundation of knowledge and concepts. Reinforcement learning is what enables that structure to be turned to a specific end. Now AI companies are using reinforcement learning in a powerful new way – training models to reason step-by-step: 1. Show the model a problem like a math puzzle. 2. Ask it to produce a chain of reasoning to solve the problem (“chain of thought”).[1] 3. If the answer is correct, adjust the model to be more like that (“reinforcement”).[2] 4. Repeat thousands of times. Before 2023 this didn’t seem to work. If each step of reasoning is too unreliable, then the chains quickly go wrong. Without getting close to co
 ·  · 1m read
 · 
(Audio version here, or search for "Joe Carlsmith Audio" on your podcast app.) > “There comes a moment when the children who have been playing at burglars hush suddenly: was that a real footstep in the hall?”  > > - C.S. Lewis “The Human Condition,” by René Magritte (Image source here) 1. Introduction Sometimes, my thinking feels more “real” to me; and sometimes, it feels more “fake.” I want to do the real version, so I want to understand this spectrum better. This essay offers some reflections.  I give a bunch of examples of this “fake vs. real” spectrum below -- in AI, philosophy, competitive debate, everyday life, and religion. My current sense is that it brings together a cluster of related dimensions, namely: * Map vs. world: Is my mind directed at an abstraction, or it is trying to see past its model to the world beyond? * Hollow vs. solid: Am I using concepts/premises/frames that I secretly suspect are bullshit, or do I expect them to point at basically real stuff, even if imperfectly? * Rote vs. new: Is the thinking pre-computed, or is new processing occurring? * Soldier vs. scout: Is the thinking trying to defend a pre-chosen position, or is it just trying to get to the truth? * Dry vs. visceral: Does the content feel abstract and heady, or does it grip me at some more gut level? These dimensions aren’t the same. But I think they’re correlated – and I offer some speculations about why. In particular, I speculate about their relationship to the “telos” of thinking – that is, to the thing that thinking is “supposed to” do.  I also describe some tags I’m currently using when I remind myself to “really think.” In particular:  * Going slow * Following curiosity/aliveness * Staying in touch with why I’m thinking about something * Tethering my concepts to referents that feel “real” to me * Reminding myself that “arguments are lenses on the world” * Tuning into a relaxing sense of “helplessness” about the truth * Just actually imagining differ
JamesÖz
 ·  · 3m read
 · 
Why it’s important to fill out this consultation The UK Government is currently consulting on allowing insects to be fed to chickens and pigs. This is worrying as the government explicitly says changes would “enable investment in the insect protein sector”. Given the likely sentience of insects (see this summary of recent research), and that median predictions estimate that 3.9 trillion insects will be killed annually by 2030, we think it’s crucial to try to limit this huge source of animal suffering.  Overview * Link to complete the consultation: HERE. You can see the context of the consultation here. * How long it takes to fill it out: 5-10 minutes (5 questions total with only 1 of them requiring a written answer) * Deadline to respond: April 1st 2025 * What else you can do: Share the consultation document far and wide!  * You can use the UK Voters for Animals GPT to help draft your responses. * If you want to hear about other high-impact ways to use your political voice to help animals, sign up for the UK Voters for Animals newsletter. There is an option to be contacted only for very time-sensitive opportunities like this one, which we expect will happen less than 6 times a year. See guidance on submitting in a Google Doc Questions and suggested responses: It is helpful to have a lot of variation between responses. As such, please feel free to add your own reasoning for your responses or, in addition to animal welfare reasons for opposing insects as feed, include non-animal welfare reasons e.g., health implications, concerns about farming intensification, or the climate implications of using insects for feed.    Question 7 on the consultation: Do you agree with allowing poultry processed animal protein in porcine feed?  Suggested response: No (up to you if you want to elaborate further).  We think it’s useful to say no to all questions in the consultation, particularly as changing these rules means that meat producers can make more profit from sel