[Cause Exploration Prizes] Software Systems for Collective Intelligence

This anonymous essay was submitted to Open Philanthropy's Cause Exploration Prizes contest and posted with the author's permission.

If you're seeing this in summer 2022, we'll be posting many submissions in a short period. If you want to stop seeing them so often, apply a filter for the appropriate tag!

In this document I will attempt to describe a cause area that is relatively neglected, or rather, unexplored, given its potential importance. The issues in question, generally, are coordination and governance. I argue that while there are continued efforts to improve traditional institutional governance, too little attention is placed on inventive efforts toward methods of focusing collective intelligence in ways that can influence traditional systems, provide coordination across traditional systems, and have the long-term potential to replace parts of existing institutional infrastructure. Such changes may in fact be necessary in order to address a set of encroaching catastrophic risks.

The Problem

Traditional forms of governance have failure modes. Democracies can fail where expertise is required for public decisions. Authoritarian systems are overburdened at the top and subject to the whims of individuals who, at any given moment, may not serve as effective leaders. Consensus and collaboration efforts can be sluggish and fail at scale. Where institutions do at times succeed at enforcing and incentivizing systems of trust, those solutions will often be confined by physical or logical borders and are constrained from establishing trust and coordination across domains.

In some sense, the problem is simple: The world is facing a set of encroaching catastrophic and existential risks and people are bad at coordination. It could be argued that the greatest coordination failure at the moment is the failure to put serious effort into solving coordination. The question then is whether or not there are as-yet-unimagined solutions that might move us toward better and more effective collective intelligence systems focused on addressing those risks. 

It seems inevitable that a technological solution to governance in general will become part of the landscape. After business, entertainment, and social, it is one of the last frontiers to remain largely offline. There may be an important opportunity at this stage to create a solution (constructed to define and accomplish EA goals) in the commons before corporate or state-sponsored solutions gain traction. 

Existing Efforts

There are a number of positive moves in this direction. The vTaiwan project moves active democracy closer to the populace and, using tools like Pol.is and others, makes consensus building more fluid and transparent.

Early efforts toward decentralized governance in the form of DAOs continue to show promise. The Gitcoin DAO  is an increasingly successful effort toward addressing a well-known coordination problem (underfunding of open source software projects). 

There are a number of academic investigations into the necessity for improved collective intelligence as a means to address catastrophic risk: Yang and Sandberg, 2022 provides a current assessment of the state of the art. Promising investigations into adaptive and resilient methods of governance include: Choi et al., 2001; Walker et al., 2010; Haasnoot et al., 2013; Kwakkel et al., 2016; Folke et al., 2010; Farmer et al., 2010. These methods are evaluated and summarized here: Fisher, L. & Sandberg, A. 2022.

Investigation Area

Networked software solutions provide an opportunity to provide new organizational structures and behavioral enforcement systems based on our best ideas around optimizing collective intelligence toward risk mitigation and positive longterm outcomes. Further, progress has brought us to a place where AI assistance towards those ends could increasingly provide useful guidance and enforcement, perhaps preempting our attempts to foil cooperative collaboration for all the usual reasons.

While there are no direct paths to replace existing forms of government with borderless software systems, there are perhaps open paths to creating enough engagement to change the ways in which a given government legislates. Consider as a hypothetical example that Facebook or Twitter were a unified voting block and lobby, its users motivated by the need for alignment rather than combative engagement. Consider also that the optics of a non-governmental body effectively addressing problems, some of which are in areas poorly addressed by government, may provide an early enabling condition for changes in existing institutions.

What might such a system look like? While a social media site with the addition of issue voting is an interesting idea, it would not be sufficiently equipped to effectively coordinate the collective intelligence of its participants. A successful coordination system would seem to need, at least, the following qualities:

• Users are impelled to address their reasoning and philosophical deficits in order to contribute and have tools available to address those deficits.
• A system in place to give users feedback on their ability to contribute (forecasting, decision making, clarity of communication, etc.)
• Users with better contribution abilities can be promoted and have increased weighting toward general decision making.
• Users with proven subject matter expertise (and sufficient general contribution abilities) are given increased weighting toward subject matter decisions.
• System structure is designed to limit the scope of relationships to a number that is conducive to collaboration (any given user's graph node count should be under Dunbar's number, for example).
• System allows rapid creation of adaptive subnetworks to respond to issues at different levels of urgency and to identify important points of intervention.
• System employs a cryptographically secure, flexible, and auditable voting system.
• Everyone welcome at some level of participation.
• The science on optimization of collaboration and collective intelligence is leveraged effectively.

Many small-scale iterations of invention and error correction may be required before a viable such system could be released in the wild. In my estimation, there are a number of variables that need to be thoughtfully considered and tested, including but not limited to:

• Graph structure
• Level of user anonymity
• Voting methods
• Evaluation and training methods
• Incentives

As an example, suppose iteration one of such an effort had the following qualities:

• A hierarchy of 5-member councils in which each user is both a member of one council and a chair of another. Top level council (level 0) is seeded with persons known to be capable thought leaders and collaborators, perhaps from EA or EA-adjacent organizations. The next level of councils (level 1) are divided into system-dictated areas of concern that match with the skills and interests of level 0 council members (acting as level 1 council chairs) and are populated with invitees from each chair. Invitations to level 1 lean toward participants with connection to existing academic/institutional organizations. Participation in deeper council levels (created as needed) will include avenues for membership from public participants of the system. Note that creation of new councils as issues move from coarse to granular could provide for a limited exponential growth. Approval processes would need to keep management of growth rate in mind.
• Public is invited to comment/vote on issues transparently deliberated by councils a la Pol.is.
• Public contribution is incorporated into voting council deliberations and decisions.
• Financial incentives are provided to encourage participation in forecasting tournaments, problem-solving collaboration games, issue-specific essay contests, etc.
• Training resources are provided to assist users in gaining greater skill and understanding of various forms of incentivized contests while integrating general instruction toward scientific thinking.
• Various forms of evaluation (results of above contests, upvoting, narrow AI classification of contributions, etc) are used to make decisions about eligibility for voting councils and council promotion decisions. Such decisions will take into account the current science on the kind of diversity required to build effective collaboration. In essence, system participants shown to be the best forecasters, problem solvers, theorists, investigators, decision makers, etc. are placed within the system hierarchy in a manner that promotes more effective collective coordination.
• Where long timelines are involved, high-level councils can create long deliberation/exploration/refinement cycles in assigned deep subnetworks (identified by areas of expertise, etc.) while more immediate concerns can be deliberated and decided quickly in pre-selected and adaptable subnetworks at or near the top of the network graph. -Note that, with time, top level councils become more populated with qualified public members that bubble up so the sense of meritocratic fairness should increase commensurately.
• A charitable fund is associated with the system and voting council members will make decisions about which cause areas to fund and how to best fund them. Existing EA organizations are encouraged to contribute to the fund and participate in the system, perhaps incentived by the need for better coordination across organizational boundaries. As end user engagement increases and participants become more directly involved with problem areas and their details, public contributions to the fund may follow.

Further iterations of the system might include, for instance, AI-powered collaboration assistance that ensures deliberations do not fall into known pitfalls that impede successful cooperation.

The existence of a charitable fund requiring distribution and direction decisions provides users an incentive to deliberate and a meaningful reason to participate. A contest structure with financial and reputational stakes provides another incentive to participate and evolve within the institution.

The above example is provided not as a solution (there are probably a thousand better starting points) but to demonstrate that it may be possible to create a system that focuses collective intelligence while maintaining something akin to the appeal of current social media sites. Participants, beyond being provided new forms of community, will be provided engagement via the ability to contribute to problems they care about. Dunking, tribalism, and sideline criticism have the potential to be replaced by the necessity to collaborate and directly face well-explicated facets of issues at hand.

Possible Next Steps

A new nonprofit organization with a board composed of knowledgeable EA generalists and EA-supportive software founders could, for a relatively low investment, commission an investigation into the design of such a system. Funding decisions around the software development process could follow from the output of that effort. 

Direct funding of such an effort by a philanthropist or a narrowly focused philanthropic organization has some potential to poison perceptions of the effort. Getting open-ended participation across organizations at the start would perhaps provide a better foundation for a successful project. 

Uncertainties

The notion of building a software-based system of governance of this kind is one that is untried. There are myriad ways to approach it and likely infinite ways to fail. Good minds would need to be put to the task of design. Even so, opportunities exist to build such a system such that:

• The wrong balance between direct participation and meritocracy creates a perception of inequality.
• Complicated and/or slow processes around decision making, deliberation, promotion, etc. inhibit participation.
• Failure to be adaptable to human issues (absences, flagging participation, unhelpful alliances, etc.) could inhibit or corrupt processes toward desired outcomes.
• Resolution processes don’t adequately address conflicts and philosophical differences at crucial decision points.
• Incentives toward participation aren’t carefully devised and pervert ultimate goals.
• System processes aren’t carefully designed to thwart improperly motivated participation (party identification, greed, ego, etc.)
• Perception issues keep the project from growing to a desired scale.
• Designed processes make scaling too slow or too fast.

If, as I suspect, the best outcomes require initially seeding the network with a chosen set of people who have the ability to foster good collaboration and make reasoned choices, then there will exist the potential for the perception of elitism. As mentioned earlier, this perception could improve as public participants earn status. In the meantime, however, a very human seeding process will be required and without the kind of computer-assisted collaboration optimization that we would expect a smart, automated system to have over time. 

Conclusion

A potential system of this kind at a large scale comes with many open questions and has many possible instantiations. Given the current set of technological tools, and given that the lack of solutions in this space could be filled by efforts with less equitable intentions, it seems the timing is good for embarking on such a project. Initial investment should be small and given the very difficult to see but somewhat possible future in which catastrophic/existential risk X might not otherwise be addressed, it makes sense to me to start the investigation.

Works Cited

Choi, T.Y., Dooley, K.J. & Rungtusanatham, M. (2001) 

Supply networks and complex adaptive systems: control versus emergence. 

Journal of Operations Management

Fisher, L. & Sandberg, A. (2022) 

A Safe Governance Space for Humanity: Necessary Conditions for the Governance of Global Catastrophic Risks. 

Global Policy

Folke, C., Carpenter, S.R., Walker, B., Scheffer, M., Chapin, T. & Rockström, J. (2010) 

Resilience thinking: integrating resilience, adaptability and transformability. 

Ecology and Society

Haasnoot, M., Kwakkel, J., Walker, W.E. & ter Maat, J. (2013) 

Dynamic adaptive policy pathways: A method for crafting robust decisions for a deeply uncertain world. 

Global Environmental Change

Kwakkel, J., Haasnoot, M. & Walker, W. (2016) 

Coping with the wickedness of public policy problems: Approaches for decision-making under deep uncertainty. 

Journal of Water Resources Planning and Management

Walker, W.E., Marchau, V.A.W.J. & Swanson, D. (2010) 

Addressing deep uncertainty using adaptive policies. 

Technological Forecasting & Social Change.

Chuqiao Yang, V. and Sandberg, A., (2022). 

Collective Intelligence as Infrastructure for Reducing Broad Global Catastrophic Risks. 

arXiv preprint arXiv:2205.03300