A big thank you to Birte Spekker, Alexander Herwix, Achim Voss, Florian Jehn, Manuel Allgaier, Peter Ruschhaupt, Rudi Zeidler, Ruslan Krenzler for commenting on drafts of this post.
Epistemic Status
Pretty convinced but with an uneasy feeling that I couldn't pass an ideological turing test.
List of threats is tentative, only meant for illustration, and probably has blind spots.
Summary
In this post, I will argue that for many EA cause areas, having secure communication and collaboration platforms is an instrumental goal. And that Privacy, security and safety aspects are currently undervalued by the EA Community and too much weight is given to how wide-spread and easy to use these platforms are. I argue that self-hosted open-source collaboration tools are a good default alternative to the proprietary cloud services provided by tech companies.
Which platforms to use is a hard-to-reverse decision in which we are trading off higher growth in the short term (1-5 years) against risks that might appear in the next 5 to 50 years.
Over longer time-spans similar trade-offs along these instrumental values may arise.
Building a good model of the risks from a lack of privacy & security seems valuable.
Introduction
From talking to different people in the community, I got the impression that quite a few people fall into one of the two following categories:
- feeling a diffuse sense of being uncomfortable about giving up their private information, but can't really justify where to draw the line.
- feeling held back by other's privacy demands, while seeing little to no risks from giving away the information in question.
I'm hoping to provide both groups with a clearer picture on the risks of lacking data privacy, so that group 2 can gain more understanding of the concerns and feel less held back and group 1 can improve their mental model of privacy and feel safer.
I'm concerning myself in this post mainly with internal communication and collaboration tools. Threats arising from debating EA topics in public spaces tied to one's main online identity are out of scope for this post.
Definitions
In the context of this discussion I will use the terms as follows:
privacy: protection of personal or confidential information from leaking out.
security: protection against deliberate attacks.
safety: protection against unintentional data loss.
data loss: no one has the data any more.
data leak: someone who shouldn't have the data, has them. This is generally non-reversible.
personally identifiable information (PII): is information that, when used alone or with other relevant data, can identify an individual.
collaboration & communication platforms: Software tools that allow groups to work together. Concrete examples are: Slack, WhatsApp, Facebook, G-Suite/GoogleDocs, Mattermost, GitHub, Asana, Trello, etc.
self-hosted (software): Software and online services that are running on computers that we control, in the sense that no actors can access the stored data without our knowledge and consent.
Why work on privacy & security now?
My general argument goes as follows
- We want to build a movement that exists for hundreds of years.
- The political landscape might change and with it the types of adversaries we might face.
- The capabilities of these (future) adversaries should inform our strategy on operational security. Especially what tools we use, and where we store our internal data.
Corollaries are:
a) internal conversations and documents stored on services controlled by third parties may become accessible to future adversaries.
b) Groups and organisations should do some threat modelling before choosing tools, deciding where to store their data and how to protect it.
c) Groups and orgs need to actually have options adequate to their threat landscape to choose from.
These lead me to believe that it would be quite useful to provide self-hosted options for collaboration software for EA groups and organisations who may lack the resources and technical knowledge to set them up themselves.
General prescriptions for what tools each EA org should use don't seem useful, each one knows their own requirements and threat landscape best.
I argue that we should investigate and document the concrete drawbacks and advantages of self-hosted open-source alternatives to proprietary collaboration platforms offered by companies. Some people might overestimate the associated costs.
Actively seeking out secure default options, preserving them as options and setting them up can help give EA orgs an informed choice when it comes to information security.
Secondary benefits are:
- Skills and means in information security will probably be valuable in several global catastrophic risk cause areas, and working on projects that provide secure tools for EA groups and organizations can help EAs build info-sec skills together.
- Managing compliance with respect to privacy regulation is easier with self-hosted platforms.
Threats change gradually
Privacy and Security should not be evaluated against today's threat landscape, but also against future changes according to their probability, and their respective potential of harm.
These changes can be very gradual, so people will disagree when a point is reached where discussions on a platform are not safe any more. Building consensus to switch might be hard, and may be a lot harder to reverse in 5 years time when the community has grown.
Do we have enemies? – Threat modelling
In this section I want to illustrate some adversaries and threats that we might face in the near term future, and what their capabilities could be.
1 Threats from state actors / governments
EAs will probably be discussing topics that some states want to censor.
They might also develop ideas that go directly against the interest of actors that are in power, or that some states consider illegal.
- The idea of building global cooperation on AI research might run counter to a states interest of "winning" the race to artificial general intelligence.
- Ideas implying the overthrow of some totalitarian regime, direct opposition to some political party, or powerful malevolent individual. (i. e. to preserve liberal democracy).
- Lobbying for new voting systems that would reduce established parties' chances of winning
Even if EAs might not openly oppose these actors, they still might funnel resources to interventions against these actors' interests.
Given Current/near future capabilities, I would argue that data stored on cloud services provided by large companies should be assumed to either become accessible to automated surveillance over the next 50 years, or the corresponding company will be forced to stop serving the jurisdiction.
Concrete threats:
Security & Privacy:
Surveillance – collection of contents as well as meta-data (i.e. who has been talking to whom) used for:
- Automatic tracking and tagging of users by interests, leading to people appearing on an intelligence agency's lists of dissidents/separatists/terrorists (terminology varies, but in general "enemies of the state")
- Imprisonment, torture, killings
Safety:
- Removal of content
- Services banning users from certain countries
- Governments banning services in their country
2 Threats from non-governmental organisations
In some cause areas, EA might be perceived to be opposing the interests of a non-state actor (group of people with some amount of power). These could be:
- Industry (for example cigarettes, animal products)
- Charities that lose funding due to EA ideas about effective interventions spreading
- Political movements that come to see EAs as the enemy (i. e. right-wing extremists might see EA as open-border-globalists, left-wing extremists might see EA as neo-liberals)
- Companies working on advanced AI may see EA lobby efforts (for example windfall clauses) as a danger to their profits
Capabilities
- Significant resources to be spent on efforts against EA
- Acquisition of smaller companies that provide discussion platforms
Concrete threats arising from this are:
Can we reduce these threats?
By involving fewer third parties (that could become compromised), and retaining control over our internal information we can prevent or reduce the impact of most of these risks.
People who deal with topics that contain information hazards probably (hopefully?) know where not to post what kind of information. But sometimes things that seem innocent in the beginning might turn out to be information hazards in retrospect or lead down a dangerous path.
Having full control over the servers on which our conversations happen is very useful in this case, and makes containing these easier.
There are clear benefits to services provided by Facebook, Google, Slack, etc.:
- They are constantly improving, and try to make working with them as simple and comfortable as possible. This leads to an improved user experience that open source tools often only imitate months to years later.
- Sharing Data and Integrating workflows is easy as long as you stay within the Systems of the given provider (i.e. Google Forms -> Google Sheets, using Facebook data about User's likes and group membership to run targeted ads, etc.)
- Due to the mentioned network effects that these companies strive for, many people already have experience with the tools, because some other organisation in their life already uses them.
I agree with these statements, and I do not value them important enough to accept the increased vulnerabilities that I laid out above.
The following are arguments against my valuation that have not yet convinced me.
I'm trying not to straw-man here, but I have an uneasy feeling that I may not represent the counterarguments faithfully. Please point out any flaws in my reasoning, and help me better understand these stances.
Side note: these are laid out in a frequently asked questions style with lots of highlighted keywords to make them more skimmable.
Everybody uses it and is fine with it
It could be argued that a large portion of the EA community is already using these services, and is therefore fine with the trade-off. This could be an example of Revealed Preferences – people saying they value privacy but then not acting accordingly.
Do we know enough about peoples' motivations and reasons to conclude this? I think there are a few other possible explanations why we would see widespread use in the Community, even if the privacy argument is fully valid:
- network effects are real and positive when you're part of the network (see lock in)
- founder effects may lead to less diversity in communication and collaboration tools (initial choices get scaled up)
- preferences seems more complicated when privacy is concerned: there are experiments pointing towards a kind of strategic information avoidance happening.
- Acemoglu et al argue that the price of my personal data may be artificially depressed because of what others with shared characteristics have already revealed about themselves, creating the impression that users do not value their privacy much.
- selection bias – unintentionally we already might have excluded the people which are most uncomfortable with having their PII on those services.
- familiarity and no motivation to learn a different tool. "My company uses Slack. I want to use Slack for EA too."
- Lack of security knowledge and experience.
Independent of the merit of the arguments for privacy, concern for privacy is widespread among academics in some countries. It might make sense for some national groups to offer privacy preserving infrastructure, even if the worry were completely unfounded. Just to appeal more to people who happen to hold these beliefs.
As EA becomes a more global community, prioritizing privacy more also helps protect visitors from more hostile places, who might not be able to use platforms that are restricted in their country.
People will leave / not become involved because of the bad user experience of the self-hosted solutions
The argument here, is that people will more gladly use a polished interface. If it makes the tasks they want to achieve easier, people will do the task more regularly and/or more people will participate.
I agree that the ease of use of the interface does matter on the margin, but way less than the central motivating factors. I. e. what is motivating people to be involved in the first place and how valuable/engaging the content is.
All else being equal, one should strive to improve user experience, but I think maintaining control over the internal information should be valued higher than user experience.
It takes too much maintenance time and effort
It costs too much valuable EA-person-time to maintain these servers. Also we might not have enough people with the skills to do that.
EA is a community that has been criticized for its skew towards tech people, and also has problems to find enough direct work to do for talented people.
To convince me that this is a severe problem, I would need to become convinced that the makeup of the EA Community changes drastically in the next years, and at the same time not enough financial resources become available to pay for professional sysadmins or at least smaller IT companies who offer managed hosting of open-source software solutions at reasonable prices.
People will know when to switch if they have something to hide
Again, I fear the process is gradual in a "boiling the frog" kind of sense. Also, evaluating this regularly has mental overhead cost: If I have a paper shredder, it might be more effective to just put all discarded documents in the shredder instead of asking yourself every time "is this confidential enough to shred?".
What if only a minority of EAs is threatened? Won't they just leave before the rest can agree to move to a different platform?
And if it's not a gradual change but a sudden change (for example a platform being banned), would we be able to get all our data out of the old platform?
How much time and resources would it cost us to get everyone on a new platform during whatever crisis that forced us to switch?
Hey, I know the people who work in/built that company, they would oppose anything nefarious happening
Arguments for this would include the project Maven scandal, which lead google to announce new principles on what AI applications they build. (Or maybe just to hide their investments in this sector under a different Alphabet subsidiary?)
A lot of things have to go right for this kind of internal control to work:
- awareness – knowledge of the project/problem has to spread, employees have to be aware of security issues
- culture – the "nefarious thing happing" has to be commonly understood as bad
- employee incentives – lots of other job opportunities, not easily replaceable
- large enough company to resist acquisition by others with different ethical standards
Note that you also need to be sure that these factors will all persist in the future.
There are just as many counterexamples where this did not work.
In all these cases, once the damage is done, there is no way back.
In a recent Future of Life Institute podcast, Mohamed Abdalla makes the argument that companies have a huge incentive to create an image of social responsibility. I would therefore expect it to be quite hard to evaluate from the outside if the conditions for internal self-regulation are in place and actually working.
We are facing a principal-agent problem here, where we are trying to verify that the incentives of the agent we put in charge of our infrastructure are aligned with our goals.
At the least we would have to do a detailed evaluation on these companies. Frameworks to evaluate a companies privacy orientation exist, but even companies that try to differentiate on high privacy standards can still suffer security breaches that are almost impossible to anticipate from the outside.
You still have to solve the same security issues, but with less resources
Yes, you have to secure each server that you set up, which needs expertise. And even Open Source Software can only be trusted in so far as you trust the people who wrote it. But you are a) a smaller target and b) in control of all future security measures.
a) target size & attack surface
The argument that large companies can better defend customers data against attacks than the customers could by themselves is sometimes called "death star logic":
"The effort behind a cyber attack is proportional to the value of breaching the system. As the value of the data held […] increases, so does the scale of its cyber threats."
In a big company the complexity of the infrastructure increases by a lot, which gives you also a larger attack surface. The supposed advantage of having a big company's resources can often be outweighed by a highly complex infrastructure that leads to leaks, security problems and outages despite all efforts.
With big companies, there is a way larger incentive for attackers to find a breach, than with some small community server with maybe 10,000 irregular users.
Small self-hosted servers will simply not attract the same attention.
b) having control
With your own servers, you can scale up security measures as the data you collaborate on or share becomes more sensitive, or if the adversarial environment changes.
Let's say a breach happens on your server. Then you have the means to actually investigate and assess the harm done, and verify afterwards that the breach can not happen again.
As customer of a company you only have press releases like these to go on:
"Our investigation found that usernames and passwords recently stolen from other websites were used to sign in to a small number of Dropbox accounts.
[…] We’re sorry about this, and have put additional controls in place to help make sure it doesn’t happen again."
– Dropbox
My argument here is that it would be easier to vet a few system administrators every few years, than the inner workings of a company.
Depending on how the threat landscape develops, this might even become it's own effort. Maybe at some point we decide to build an EA-ops-sec Organisation that provides security consulting for other EA organisations.
Self-hosting does not solve the surveillance problem
Yes, your cloud provider could also be forced to give a state access, or an intelligence agency might exploit a software bug. Even warrant canaries probably don't help.
Finely grained surveillance efforts are a lot more costly to state actors, and won't be used as routinely. Only high value targets for intelligence services get special treatment.
Meanwhile on Facebook, Google, Slack, office365, etc. one should assume automated snooping efforts at least by five-eyes intelligence services.
Should it be revealed that your provider is not safe any more, it is easy to just move your virtual server to a different jurisdiction. Probably without the users even taking notice.
For communication the medium-term goal should be end-to-end encryption solutions with perfect forward secrecy, as they would prevent all third-party access.
You're overvaluing this because of your own availability heuristic
There is some evidence for this:
Many of these concepts may be more present to me, because I used to work with Colombian human rights defenders who frequently were targets of surveillance, threats and violence. I saw many examples of people ignoring good operational security practices due to convenience and there not being secure defaults.
Also setting up servers for collaboration tools is a big part of my day job, so I perceive it as less complicated than the average person.
So yes, maybe I am just used to doing security analysis in this adversarial way, and maybe I see less drawbacks to self-hosting.
To convince me that I am overvaluing this, I would need to see that the increased community growth through using more widespread software services with marginally better user interface/experience largely outweighs the expected long term harm to the movement through adversaries gaining access to our internal information.
Prediction Time!
(sadly I could not figure out how to directly embed these using the markdown editor, and the regular editor breaks my footnotes)
update: You can now vote on these in the comments below!
-
Will internal communications of an EA organisation be part of a data leak by 2040?
-
Conditional on all EA community building Slack Workspaces being replaced with Mattermost instances, would user participation in those spaces drop by more than 5%?
-
Will an EA aligned organisation become target of directed government repression by 2040? (for example withdrawal of operating license, denial of work visas, etc.)
What I would propose
When choosing tools for collaboration and communication, I'd recommend to default to:
- collect the least amount of data that still allows us to full-fill our goals
- not give any data to third parties, because this may become irreversible
When selecting tools we should justify why we deviate from these defaults. (I expect it to be justified in many cases.)
What kind of threat analysis can we do?
Ultimately the depth of analysis is up to each organisation. A useful toolbox for risk assessment in this area is "How to measure anything in cyber security risk" by Douglas W. Hubbard and Richard Seiers. They also provide example spreadsheets on the book's website.
Systems-mapping is also a useful starting point to identify relevant actors and possible vulnerabilities.
As threat levels increase, other tools from the operational security toolbox (such as red-teaming) can be useful.
TL;DR plz
We may not have anything to hide yet, but should take precautions now, because in the near term future many of us probably will.
I upvoted the post because I thought these were worthwhile ideas to bring forward, and because I love the use of Elicit predictions. I also appreciate the attention paid to the idea that different groups/orgs will have different security needs.
However, I think some of the arguments in favor of "tech company platforms" are considerably stronger than the author does -- and that platform-switching has higher costs than the author may believe.
I'll share the examples I have time to give:
I think it's very easy to undervalue convenience. (Bryan Caplan gives examples of this, though his overarching point isn't really related to this post.)
I've heard many people tell me that their Forum participation sharply increased or decreased because we added a convenient new feature or ran into a bug that affected their work in a relatively minor way. Loading times can have dramatic effects on how people use websites. And while I haven't seen statistics on this, I'd guess that communities often shed lots of members when they migrate between platforms; building a new app and interface into your existing routine/habits isn't always trivial.
Good content is almost certainly the most important thing, but that doesn't mean convenience isn't also important, or that we can decide it doesn't matter very much because it's not the central motivating factor in how people participate online.
EA might have a lot of technical people, but that doesn't necessarily translate to having lots of people who are willing to volunteer as sysadmins, will do the job reliably, and will transfer necessary information to a new person if they have to step down. (Google may be more vulnerable to breaches, but it seems much less likely than a small private server to lose data because someone makes a technical error or loses a key password -- though I don't have much confidence in this assertion, since I'm not a very technical person.)
As you said, we don't just have to use volunteers -- many EA orgs are well-funded, enough to pay for this kind of service themselves. But others already have to scrimp and save to run their core programs.
And it's really nice that our current systems can be run by nontechnical people -- it's one less barrier to starting a new group or project, even if you could track down a volunteer sysadmin given time.
As you say in your post, different organizations will have different requirements in this space. Many orgs probably should be taking secure communication more seriously. But maintenance time and funding are going to be real limiting factors for many orgs (especially smaller ones), and I think that's a serious consideration.
This seems like another way of saying that you have a strong prior in favor of valuing extra marginal security more than the convenience we'd lose in order to achieve that. That's perfectly valid -- but I don't think anyone's likely to soon be in a position (capability + willingness to spend time) to get good numbers on the expected difference in community growth in two hypothetical worlds. So you'll have a hard time finding arguments that are capable of meeting a standard that would convince you.
In cases like this, I like to ask myself whether there are more modest/achievable types of evidence that could shift my position. One useful resource for gathering such evidence is the EA Polls Facebook group. You could consider asking questions like:
Questions like these can help you build better estimates of "increased community growth", without forcing you to put an entire model together all at once.
+1 on convenience is also important
I think there is a good chance that many more people would participate in NEAD discussions if NEAD had started with Slack rather than Signal first and then Zulip. Even with Mattermost (arguably more user-friendly & intuitive than Signal and Zulip), there might be differences. Mattermost still uses markdown formatting, and this does not seem intuitive for non-technical people (I found it confusing at first and now I still find it quite inconvenient compared to extra buttons or keyboard shortcuts like "Ctrl + B" for "bold").
Maybe we could survey everyone who has a profile on Zulip now (and ideally also those who were invited but declined) and ask them if the platform choice had made a difference for their engagement? (though that would only yield data for Zulip, not for Mattermost)
Thanks Aaron for sharing your forum experience, that's useful, and also for your other thoughts!
Quick meta note: Would it make sense to put different arguments in separate comments? That would allow others to upvote your arguments specifically, instead of only upvoting the entire comment while they might only agree with 1-2 out of three points you made.
It might make sense! I'll try to keep that in mind.
Pros: Easier to discuss and vote for separate points
Cons: Lots of separate replies and replies-to-replies means lots of notification and a variety of different threads to track; there's a risk that I'm too lazy to split up comments like this
I strongly doubt that a poll hosted on Facebook will provide unbiased evidence on those questions.
Thank you Aaron for the thoughtful reply.
I find your suggestions on better questions to get more achievable types of evidence very useful. @Manuel_Allgaier and me will ask them or similar ones on the EA Berlin Slack, a German EA Telegram channel and in the FB group you mentioned.
Yes, that is a good way to rephrase my position.
This is probably a central crux for some. If you came to believe that the risk of data loss through admin error on a self-hosted system were lower than the breach-risk at Google, would that change your view on the convenience-security trade-off?
I don't think it's about total likelihood of an event, but expected impact of said event. And because I have very weak priors about the likelihood of either event, getting any new information would probably change my view about the trade-off in some way.
But changing my view on the trade-off looks more like "I now think EA funders should be open to spending $X per year on this rather than $Y" or "I now think groups with risk profile X should now be willing to switch even if their activity drops 10%", rather than coming to believe something more sweeping and conclusive about the entire topic.