I'm not sure naive total utility maximization [in a static framework] is the best framework to be thinking about dealing with existential risk over time.^[1]

Assuming the number of risks and error bars are not trivially small, the universal outcome of concentrating all your risk mitigations on one is that most risks continue to be a high as they could possibly be. The modal outcome is that the risks ignored includes at least one risk greater than the one all efforts are concentrated on  mitigating. Some reasonable assumptions in the article above show this can hold even where the actual biggest risk is orders of magnitude greater than the one targeted. In the diversified approach, less money are devoted to reducing the perceived biggest risk, but the rest is apportioned to reducing other risks. This seems more robust to conventional assumptions like uncertainty and some risks being easier to mitigate than others.

1. ^{^}

   And tbh I'm not even seeing an average utility boost from concentrating on the single largest risk as opposed to mitigating lots of risks without ancillary assumptions like increasing returns to risk reduction expenditure or the actual value of many risks under consideration being 0.

It would be interested to see a more detailed and systematic report on the activity and findings so far.

In some respects, it seems like a strange thing for GiveDirectly to be piloting. On the one hand, GiveDirectly has expertise in systematic studies of behavioural change in LDCs , and the chatbot possibly also performed programmatic functions in a cost effective manner. On the other hand it involves a charity known for its "let local people decide how to use money spent on their behalf, Western aid agencies doing it can be disempowering and often wrong" ethos asking "which parameters should we use to fine tune this [adaptation of a commercial] product we've designed to give them the most suitable answers before scaling up its deployment"... which seems like a very different ethos and approach.^[1] 

The conclusions highlighted from the research so far - both that if you give poor Rwandans access to ChatGPT they have a similar range of interaction to other humans^[2] and that responses generated by an LLM with no meaningful local training dataset were often inadequate - seem unsurprising. I am sympathetic to arguments that people make better decisions with access to information, but I am also sympathetic to arguments a ChatGPT derivative is not the most valuable information Rwandans could receive (and may have minimal or even negative value)

I'm not actually sure what the costs of acquiring relevant local data and training a chatbot to achieve greater fluency in spoken Kinyarwada dialects and safeguarding against advice that is very bad in a local context are,^[3] but they seem like a pretty relevant benchmark, since they might actually be considerable on a per user basis and the alternative for critical information like "what is the nearest health centre" might be something like signing people up to email lists, or a small number of human agents in Kigali costing surprisingly little.^[4] I guess there's also the "who's paying?" question, especially when the current implementation appears to involve providing training data for one of the world's most valuable companies (and obscure languages may or may not add value to their model). 

I feel one relevant benchmark for GiveDirectly specifically might be "what is the estimated cost per per person reached to improve it: would locals rather have a better chatbot or the cash?". It's possible the insights they're getting are extremely valuable particularly in the context of limited/no of web access, but it's possible they're not...
 

1. ^{^}

   the relevant comparator might be the One Laptop Per Child project. Well intentioned, theory of change centred on the idea that people in LEDCs can be empowered by interacting with modern technology and better information too, but perhaps actual educational benefits didn't really stack up with the costs and the participants would have chosen to have something other than a computer

2. ^{^}

   I must admit, I am curious about the extent to which Rwandans engaged in "witty banter" or attempts to manipulate the chatbot into saying something silly...

3. ^{^}

   I don't know how bad the speaking and dataset is, and whether an adequate "solution" looks like a finetuning prompt with some info or developing a corpus of services data and synthetic idiosyncratic Kinyarwada to fix the model, but the latter option could be very expensive compared with the people it would actually reach...

^{^}

I suspect you get many person years of Rwandan human call centre time for a month or two of a mid-level AI engineer's time...

Show all footnotes

yeah, agree there are some military subsystems that would go down (and alternative PNT options have their own drawbacks). But it's not a particularly decisive advantage: to take the topical example if Russia took out global GNSS it wouldn't seriously harm the defensive posture of Ukranian front line troops that have been experiencing local GNSS jamming and finding alternative ways to get drones to work for years now, and would disrupt a bunch of Russian based logistics and civilian systems further from the front line nearly as much as it did to Kyiv's. So it wouldn't even significantly help them advance in Ukraine, even before the likely response from NATO and China was considered...

I think the conclusion that diversification is a good strategy follows trivially from the optimizers' curse: if you focus all your efforts on the apparent biggest threat, you've probably just focused on the cause with the largest risk assessment error and entirely neglected the actual biggest threat. A more diverse allocation is more likely to address the actual biggest threat. If there are diminishing returns to resources allocated to mitigate particular risk areas that makes diversification look better (complex nonlinear returns complicate it). As does the possibility that larger errors in risk assessment for a particular type of risk are inversely correlated with ability to invest in the best mitigation strategy for that type of risk.^[1]

But your point about adverse selection is a good one too. Metrics are gameable, and there are stronger incentives to do so when funding is "winner takes all" rather than "we disburse funds to a wide selection of causes and value rigour and disclosure of uncertainties"

1. ^{^}

   I think there are probably exceptions to this, but I think it's generally true. Good understanding of celestial mechanics and early warning systems, for example, are absolutely essential to potentially preventing hypothetical large space rocks colliding with earth, but also mean that we are less likely to overestimate the imminence of destruction by a rogue asteroid than we are for more unpredictable phenomenon.

A few comments, some of which you may be intending to cover in updates

• First of all we actually have a pretty decent idea of what happens in GNSS-denied environments because localised GNSS jamming is a thing,^[1]  It's especially a thing in combat zones, which means that  people and infrastructure affected typically have other problems^[2]
• Because GNSS denial is a thing, militaries have alternative PNT systems to aid them in combat. So actively disabling GNSS satellites is a pretty extreme measure that mostly hurts civilians, including in about 250 countries not currently at war with you. And if it involves use of anti-satellite weapons or EMPs, probably takes out a whole bunch of other space infrastructure too^[3]
• As it's a pretty extreme measure that annoys everyone worldwide without even offering you a decisive advantage in a local conflict, it's most likely to happen during escalation of a great power conflict. Great power conflicts mean that sectors like maritime would be experiencing COVID level downturns already. The "solar storm" is more interesting because it might be largely unexpected (and would also likely impair a lot of non-GNSS comms stuff)
• But costs of nuisance level GNSS jamming in borderlands between states not actually at war (say, the Baltic...) has a scaled down version of this impact which I guess might be underestimated...
  
  
  Interested to see the followup

1. ^{^}

   Similarly, people working with navigation systems prone to spoofed location results (maritime navigation) have to find workarounds

2. ^{^}

   though electronic warfare can affect neutral neighbours and overflying aircraft too...

3. ^{^}

   current generation GNSS satellites are in fairly empty medium earth orbits and can be disabled without kinetic weapons, but this doesn't rule out collateral damage and likely won't be the case for future GNSS (in part because they want more redundancy even though the system(s) just work.

Above all it implies don't focus the vast majority of efforts on one cause.

That might not be practical for career choices,^[1] but it's certainly possible for a funder or movement

1. ^{^}

   though a corollary of it is "don't assume that just because you've picked direct work that your career choice is maximally good and stuff like donations and helping others is just a distraction". This is arguably true for speculative career choices even if the optimal cause is the correct one (i.e. even if AI x-risk really does dominate everything, lots of the promising approaches to resolving it that people might choose will have no impact)

Other than OpenAI or Anthropic, I don't see an AI company shutting down being taken remotely seriously by anyone outside a very small number of people that understand how that lab was performing, most of whom already have their own very strong views on AI safety. To most people, it just says "loss making entities coming up with elaborate excuses for AI bubble starting to burst" (or "I told you Elon was full of shit " or even "look, a European/Chinese lab cannot compete with the US AI innovation because their governments have forced them to think about alignment too much, let's not make the mistake of regulating...")

OpenAI and Anthropic doing it would at least cut through to the average person/policymaker. But even if you believe they're sincerely mission driven and owe their stockholders nothing, OpenAI and Anthropic are not IPOing this year because they believe that shutting down is the correct course of action

Yeah, I think that's true on a lot of politics. Just think that many millions have been spent to make "actually international aid is really important both for saving lives and US soft power" an unfashionable argument in Republican circles, and Big Aid already has very good lobbyists working on the Dems too.

An interesting question is whether there's much more scope in a European context, where questions about foreign aid are driven more by budget constraints and less by partisanship and xenophobia and "might the money be spent better on filling gaps in PEPFAR etc than current policy" is an argument policymakers might not be hearing so much from existing aid lobbyists. 

Seems that's because there's significantly more scope to diagnose (rightly or wrongly) communication errors between people, who have motivations, expectations and emotional reactions, belong to cultures and hierarchies and communicate with subtext...