Should EA Global London 2021 have been expanded?

I think it's great that CEA increased the event size on short notice. It's hard to anticipate everything in advance for complex projects like this one, and I think it's very cool that when CEA realized the potential mistake, it fixed the issue and expanded capacity in time.

I'd much rather have a CEA that gets important things broadly right and acts swiftly to fix any issues in time, than a CEA that overall gets less done due to risk aversion resulting from pushback from posts like this one*, or one that stubbornly sticks to early commitments rather than flexibly adjusting its plans.

I also feel like the decision not to worry too much about Covid seems correct given the most up-to-date risk estimates, similar to how conference organizers usually don't worry too much about the risk of flu/norovirus outbreaks.

(Edit - disclosure: From a legal perspective, I am employed by CEA, but my project (EA Funds) operates independently (meaning I don't report to CEA staff), and I wasn't involved in any decisions related to EA Global.)

* Edit: I don't mean to discourage thoughtful critiques like this post. I just don't want CEA to become more risk-averse because of them.

Note/disclaimer: I recently started working on the Events Team (I do communications and impact assessment), and I shared this response with them before posting it (for edits and comments). Still, the thoughts here are mine.

The tl;dr:

I think you bring up some good points. I agree that we could have planned further in advance, been more transparent, and done more to reduce COVID risk at the event. I think these are important points, and I’ll discuss them below.

But in the end, I think it was right to increase the size of EAG: it roughly doubled the value produced by the event, and I think the increase in COVID risk was small compared to this benefit. I also think that there are some errors in your post. Finally, I want to note that our survey data suggests most attendees were happy with how we handled this situation.

So let’s get into it.

We should have put more effort into anticipating the quality of applications than we did (but we’re also not sure if we could have done that much better)

One of the claims that you make in this vein is:

• “Expanding after the fact was substantially worse than deciding on a larger conference from the beginning.”

This is true. Expanding capacity so late into the whole process surprised our attendees and speakers with last-minute changes and overwhelmed the team, impeding our ability to perfect various elements of the conference. You accurately note that “only deciding to expand the event a month in advance reduced CEA's (and other actors') ability to make changes to the event to compensate for the increased size.”

The key question, though, is whether we could have realized that we would need to make this change sooner. So on to your other point:

• “The decision to expand was based on unsurprising information that should have been taken into account in advance.”

I think this is somewhat true, although I also think hindsight is particularly strong here. There were a number of difficult-to-predict factors at play. For instance, we provided more funding and support (for travel and accommodation) than we had ever before, which we guessed would increase attendance. But on the other hand, we thought that COVID caution might reduce the number of applications. I don’t think we had great data on how these factors would balance out.

Finally, just to clarify; the major surprise was not the number of applications, but rather their quality, which became clear after applications opened in September. In retrospect, we wish we had opened applications sooner.

Still, we should have spent more time trying to predict the number of great applications we’d get, and we generally could have been more careful with that process.

So I agree that in the future we should do better modeling, use forecasting for our planning, and be more aware of broad changes to the community. We’ll try to make explicit plans for being better at this in the future, and we welcome advice on how to do this.

It would have been better if we were more transparent about our reasoning

“The decision to expand the event was underjustified and untransparent.”

I agree.

To be clear, though, we were not trying to be untransparent: we were just busy with other things and didn’t prioritize this. At that point, it seemed that making sure that the schedule would make sense, that volunteers would know what to do, or that registration would happen was more urgent than writing up and publishing our thought process.

But I agree that by not doing this, our decisions may have seemed mysterious and underjustified. We didn’t explain what had changed since our 2019 posts, and this may not have been the right choice.

We made mistakes when attempting to make EA Global as safe as possible for attendees.

“CEA undercompensated for the increase in risk.”

Here are some things we did to make the event safer:

• We required all attendees to be fully vaccinated (except a few minors). We checked the proof of vaccination.
• Guest-facing external contractors were vaccinated. In some cases, vendors informed us we were legally prohibited from asking about their staff’s vaccination status; in those cases, we spent money and effort to make sure that they were taking regular lateral flow tests.
• We made over 1000 lateral flow tests available for free to attendees, and required that attendees do a lateral flow test before attending (on an honor system).
• We had thousands of masks onsite (though most attendees chose not to use them).
• We issued refunds for anyone who felt sick even if they got negative tests.
• We asked the venue to increase ventilation and to keep the windows open (see below for what happened here).
• (Here’s our COVID protocol.)

We also offered refunds (including for travel/accommodation) to everyone who signed up before we expanded the event, to allow people who weren’t comfortable with the larger event size to change their plans.

We considered some further measures, like checking that everyone completed tests every day. After assessing the risks and consulting with our advisory board, we decided that the benefits of verifying these (rather than trusting people to carry out tests as we asked) didn’t justify the significant costs.

But we made some errors here. For instance, we asked the venue to keep windows open, but they didn’t do this as much as we wanted them to. I think we should have pushed harder during the event for them to do what we had requested.

A note about how COVID risk relates to the decision to expand the conference. According to our model of COVID risk, the key factor in risk of infection is how densely packed attendees are. As we doubled the capacity, we also approximately doubled the size of the space available for lunch (and made sure that attendees actually spread out into this space) and other sessions. We did this by reserving a second venue (a good chunk of the Barbican) and adding the marquees for one-on-ones. So I think that although the risk increased a bit, it didn’t increase by that much (substantially less than by 100%).

The decision itself

It’s September and we’ve realized that the quality of applications we’re getting is far higher than we expected. We’re faced with the decision described above: do we grit our teeth and stick to our original plan, or increase the capacity two-fold?

When I first heard the proposal to expand the conference size by around 100% (the first proposal was actually a bit smaller), I thought it was insane. I imagined the chaos during registration, the confused emails, and, indeed, the increased COVID risk. We started brainstorming ways that might help us expand capacity without increasing COVID risk, and came up with the marquees and the second venue. I still thought the event would end up worse.

But then during a conversation with Amy, I realized that I was thinking from the point of view of a single attendee— someone admitted in the first wave of applications to an event for 500 people, who would suddenly be introduced to a more confusing conference for 1000. I could viscerally feel the loss in quality.

What I wasn’t imagining or feeling was the situation of an attendee who could only attend if we expanded capacity. They would be stripped of the chance to solicit feedback on their new animal welfare initiative, recruit candidates for their growing AI safety startup, get feedback on their big career decision, connect with potential funders or collaborators, or get a better understanding of the most pressing issues in the Global Health and Wellbeing space. This lost impact was invisible to my System One, but it would nevertheless be a massive amount of lost impact.

In other words, I realized that even if the conference were a bit worse for those who had already registered, it might roughly double the total value, by helping more people to attend.

In order to fulfill our obligations to the people who already registered, we offered full refunds (including travel expenses) to anyone who did not want to attend the larger event. (Although I acknowledge that psychological lock-in effects are real.)

I think that we managed the increase in capacity well, and we didn’t significantly worsen the experience relative to a 500 person conference: logistics were smooth, COVID risk wasn’t much higher, and they got access to nearly 500 extra people. In fact, due to the increased value from access to the extra attendees, I think the change might have been net positive for the initial 500 attendees (but we’re not sure overall).

In hindsight, I think the decision was correct.

Survey data

Here is some data from the feedback survey we sent to in-person attendees to back this up.

We asked attendees whether they wanted fewer or more attendees (a multi-part question introduced as “What changes should we make for future conferences?” where the options were "N/A" or "No strong opinion", Fewer, Keep the same, More).

Here is the result (587 responses):

We also asked “How satisfied were you with pre-conference communication?,” Of 596 responses, here are the results.

We asked “How satisfied were you with our COVID policy?” Results (593 responses):

[EDIT: Note that we plan to follow up with people who applied and were accepted but could not attend or cancelled their registration, and their responses to the above question may be different from those of the people who did come to EA Global. I’m guessing that in aggregate, the effect will be small due to the numbers (most people who were accepted did attend), but I will also consider those cases separately.]

We asked “How satisfied were you with COVID safety at EA Global in practice?” 595 responses. Result:

The average response for “How likely is it that you would recommend EA Global to a friend or colleague with similar interests to your own?” (on a scale from 0-10) is 9.11. For reference, this was 8.2 for EA Global: San Francisco 2019, 8.5 for EA Global: London 2019, and 7.8 for EA Global: Virtual in 2020.

I don’t add all of this here to claim that we did everything perfectly. I don’t think we did, and we plan to interview attendees with different perspectives to figure out how exactly we should improve in the future, both with respect to COVID and with respect to other aspects of running conferences.

But I do want to say that we took the decision to expand the conference (as well as the COVID implications of this decision) seriously, and I think it was the right decision.

In any case, I want to thank you for posting this on the Forum. I’m sorry for how these changes affected you, and I think it’s good for these criticisms to be discussed publicly.

Maybe it is important for folks to know that I have personally relaxed a lot about COVID risk. I think for most people in our target audience/demographic (young vaccinated people) the value of attending EAG is worth the risk. This is not because I think we create a space with near-zero risk of COVID exposure, but rather, as I say below, COVID is not the bulk of the risk/cost of this event. 

I don’t make EAG decisions based on my own models alone, which is why I consult community leaders and the COVID Board, and why we encouraged attendees to do their own calculations and decide their own risk tolerances. I expect people shouldn’t come if they are particularly cautious and (as you stated) some didn’t. (As a parent of unvaccinated young children, I think about my risk tolerance differently than I expect a typical attendee does - I understand this is a personal thing that will vary by people’s circumstances.) I’m glad people could decide for themselves whether the value was worth the risk (many chose to attend, some didn’t). Perhaps I should have been more explicit about this.

I am not especially concerned about Covid risk (and am not convinced responsibility for evaluating Covid risk should lie with CEA to the extent this post sometimes implies) but I think these criticisms are pretty reasonable and I appreciate seeing them communicated so clearly.

I agree in particular with 2. I'd have liked to see some modelling that could be criticised. I don't think I want more processes here because those will probably clog things down in future, but I'd like to understand what caused decision making to be be hidden and hard to correct here.  Were the CEA team understaffed/under-supported?

I'll add that afterparties were unmentioned but clearly higher risk than the events in general and very predictable that they would take place. Some thought/modelling here would have been nice too.

Likewise it feels that if we can't deal with risky(?) events well as a community, who can? And if we can't do things like this well, I am concerned about our decision making around complex risk in general. 

Thanks to the CEA team who ran a great event.