This is a short writeup of a June 3-4, 2024 workshop held by the Astera Institute and Metaculus called Scale is All You Need, which focused on advancing the field of synthetic biology. The two-day event brought together domain experts with the hope of identifying a set of technical innovations and scientific insights that, if developed, would help synthetic biology companies scale to the point that they can compete with traditional products. You might find this document useful if you are interested in synthetic biology or collective decision making processes.

You can forecast on a modified subset of the questions created during this workshop here.

Participants broke into groups to consider how to advance subfields like:

• Biologics — therapeutic molecules produced by living cells
• Improved crop yields & nutrition
• Plant-based proteins
• Carbon capture & storage
• Biomaterials — materials engineered to interact with biological systems
• Clean chemical production

To organize these activities, the workshop used a novel framework for collective sensemaking and decision-making called the Delphi-Hall Loop^[1]. Developed by Cate Hall and the Metaculus team, this approach builds upon and extends a well-established method known as the Delphi Process — a technique originally developed by the RAND Corporation in the 1950s to estimate the proliferation of nuclear weapons. 

Much like the Delphi Process, the Delphi-Hall Loop asks multiple rounds of questions to subject matter experts to understand and refine their beliefs and their reasons for holding them. Unlike the Delphi Process, the Delphi-Hall Loop is about more than just explaining what’s going on, it’s about then using those insights to guide action. The Delphi-Hall loop is structured around four main activities:

1. Defining objectives
2. Identifying bottlenecks
3. Proposing interventions
4. Forecasting potential impacts

The workshop served two main purposes:

1. Generate useful ideas for synthetic biology research
2. Test the Delphi-Hall Loop as a tool for collective sensemaking

Following the workshop, the Astera Institute used insights from the process to inform their research fellowship funding decisions. This practical application demonstrated the potential usefulness of the Delphi-Hall Loop in particular and forecasting in general for addressing complex scientific questions.

Overview

The purpose of the Delphi-Hall Loop is to enable a community to identify the best actions to take to achieve a goal. Each phase of the loop addresses one question.

• Objectives — what do we want?
• Generic example: “Improve access to clean water.”
• Bottlenecks — what’s holding us back?
• Example: “Waste disposal practices.”
• Interventions — what can we do about it?
• Example: “Offer community courses on waste disposal.”
• Forecasts — what difference will it make?
• Example: “If we offer waste disposal courses, how much will it improve access to clean water?”

Each phase consists of two stages. The first is a generative stage, in which participants enumerate as many ideas as they can. (I.e. in the Objectives stage they come up with as many objectives as possible.) The second is an elisive stage, in which participants select the single (or few) ideas they want to take to the next phase. (E.g. if one of their suggested objectives was, “Improve access to clean water.” they might choose that as a target for understanding the bottlenecks to accessing clean water.)

Astera Workshop Example

In Astera’s Scale is All You Need workshop, in addition to following the general Delphi-Hall Loop, we also added additional structure to the process by giving participants access to two spreadsheet templates to fill out and a Metaculus instance for forecasting.

Here’s an example of sample outputs from the Scale is All You Need workshop, following the group tasked with advancing plant-based proteins (protein synthesis in plants to act as a substitute to animal-based proteins).

Objectives for advancing plant-based proteins:

• Make agriculture more climate-resilient
• Restore and enhance natural systems
• Improved crop yields & nutrition
• Increase rate of development and drug approval and accuracy of those new therapeutics actually perturbing the right targets to meet primary endpoints more often

Objectives are the goals that the participants agree are worth addressing. Objectives were captured with a survey sent out ahead of time and were then clustered into themes. Experts were assigned to theme groups. During the workshop, a short amount of time was provided to add additional ideas to the list and to select the most important objective to address. Note that objectives are better the more specific and measurable they are.

Selected objective: Make agriculture more climate-resilient

Bottlenecks to making agriculture more climate-resilient

Bottlenecks are the impediments to achieving the selected objective. For this, we asked each participant to generate bottleneck suggestions, and then to (blindly, without looking at others’ inputs) estimate the average size of the impediment to the objective, and then sorted them by the largest mean impact. The ability to see one another’s scores in the final step permitted participants to discuss the difference in their estimates and provide adjustments. This part of the process worked pretty well. From those, the most impactful bottlenecks were chosen.

Selected bottlenecks:

Interventions on key bottlenecks

Interventions are the actions we can take to unblock the bottlenecks. Participants were asked to generate interventions to address each bottleneck, to rate each intervention on the dimensions of impact, feasibility, and novelty. A simple composite score was calculated, which prioritized impact, then feasibility over novelty. These intervention recommendations were passed off to the forecast question phrasing team, who turned them into conditional forecasts.

Forecasts on key interventions

Forecasting is using information and reasoning to provide educated estimates of future quantities. Skilled forecasters have been found to outperform even subject matter experts; in part this is due to their ability to consider the whole picture and relevant base rates.

To equip our participants with forecasting abilities, we brought in several forecasting experts to teach the basic skills. Then, participants were asked to forecast on conditional questions.  Conditional questions ask, “if we achieve this, what will happen” and then “if we don’t achieve this, …”. The benefit of conditional forecasting is that it provides an estimate of the size of the impact if you achieve the goal. For example, you can see in this example the forecast that open source tools for protein production will have a significant impact on making alternative proteins competitive in the market.

If you’d like a copy of the agenda we used for the conference please get in touch.

Learnings

We learned quite a lot with this experiment. Here are some suggestions for ideas you might try or priorities to keep in mind if you plan to run a similar workshop:

1. The expertise and interest of participants matters quite a bit. Make sure to make it easy to join the groups and topics where attendees have the most interest.
2. Consistently, the highest value that participants report from a conference comes from the people they meet. Permit lots of mixing of groups.
3. In subsequent experiments, some participants reported they would like to do more frequent and more lightweight forecasting. For example, “I’d like to forecast the value of completely removing this bottleneck.” Consider teaching forecasting earlier, and providing opportunities to forecast.
4. In this Astera conference, we asked participants to estimate impact, feasibility, and novelty for interventions on key bottlenecks, but you may have different criteria you care about. One idea, similar to the above, could be forecasting the likelihood of success—and then the impact of the intervention if there is success.
5. Perhaps, after forecasting, create a phase where participants select their single most promising intervention and present it to a panel of judges for a significant monetary prize.
6. More time spent on iteration and refinement of the selected forecast questions would likely have been worthwhile. Participants didn’t have previous forecasting experience coming into the workshop, and this resulted in less useful questions.  

The Astera Institute plans to publish a more formal report on the workshop’s outputs. We will look forward to sharing it. 

Get in touch if you want to discuss getting access to your own Metaculus instance. You can also find the spreadsheet templates used for this workshop here and here. We encourage you to experiment with your own approaches to structuring collective decision-making processes like the above.

FOOTNOTES

1. ^{^}

   The Delphi-Hall loop is named after Cate Hall, who originated the language of objectives, bottlenecks, and interventions central to the process. The Metaculus team adopted and incorporated the loop into subsequent workshops. (Hall has suggested that the naming may overstate her contribution.)