EA Global London 2021 took place last week (29th-31st October 2021). As with all EA Globals, it was in most respects an amazing conference, far above the standard I expect from other similar events. However, it also took place in the midst of a major COVID wave in the UK, with major concerns about vaccine escape and long COVID (including for vaccinated people). Many people I spoke to were concerned about the COVID risks posed by attending the conference, and I know of at least a few high-value members of the community who declined to attend for this reason.
The event had been cancelled or rescheduled multiple times in the past, and this iteration was originally intended to be relatively small, about 500 people. However, after many more people applied than there were places available, the decision was made to roughly double the size of the event. This change was made relatively last-minute, being announced about a month prior to the conference. I believe this event was the largest EA Global to take place for a number of years.
I was concerned about the decision to increase the size of EAG, and especially unhappy with the last-minute nature of the change, the low level of transparency around the decision process and reasoning involved, and the lack of apparent compensation mechanisms to reduce COVID risk in the face of so many more people. When I shared an earlier (and more aggressive) version of this post with CEA prior to the conference, I received a number of corrections, clarifications, and counter-arguments – but almost none of these were in the public domain. Even though the conference is now past, I think it's important to have a public conversation about these issues, and perhaps help avoid similar issues with future events.
At present, I am highly uncertain whether or not the event should have been expanded last-minute as it was. At the very least, I think it's not-blindingly-obvious that this was the correct decision, and if CEA had good evidence that it was I strongly believe they should have shared it publicly (and should still). In general, my discussions with CEA about this have updated me about the situation in some important ways, but I still believe that many of my arguments below are correct:
- [Medium confidence] The decision to expand was based on unsurprising information that should have been taken into account in advance.
- EA Global conferences have been routinely majorly oversubscribed for years now, especially since the decision was made some years ago to deliberately reduce their size. Given this, the fact that there was a large waiting list for this conference was not at all surprising; I think I would have given an ex ante probability of over 90%. I'd be very surprised if the events team wasn't also expecting this.
- The feedback I've received from CEA on this point is that, while the mere fact of a waitlist wasn't necessarily surprising, the quality of the people in that waitlist was much higher than anticipated. This is consistent with the various metric updates from CEA's Q3 updates; however, as such I think it should still not have been hugely surprising (though hindsight is of course 20/20).
- (I previously had some discussion of COVID news in September here, but I've cut it since it wasn't raised by CEA as a major factor in their responses to me.)
- Based on communication with CEA, it is also my understanding that many of the most promising applicants only applied after the initial early-bird deadline – and thus after many places at the conference had already been allocated, making the need for expansion much more pressing. I don't see any reason to doubt this – but it suggests to me that CEA's systems for deciding how and when to allocate conference places was quite badly flawed.
- [High confidence] The decision to expand the event was underjustified and untransparent.
- As with most COVID-related decisions made by the CEA events team over the course of the pandemic, the change in policy was justified solely by an appeal to authority – namely, that they consulted with their COVID advisory board. Although my understanding is that more people were consulted, no further justification or discussion was made available.
- In particular, no estimate was given of how much this change in policy would alter the risk of the event – the expansion email advised attendees to use microCOVID to help estimate their risk, but didn't provide any modelling or estimates they had made themselves as part of making this decision.
- When one prominent Forum user (Saulius) asked for estimates of the risk, this went unresponded to for over a week. And in my discussions with CEA over this issue, I have yet to be linked to any pre-existing in-house models or cost-benefit estimates CEA used to make this decision.
- In general, it's currently my impression that CEA did relatively little in the way of in-house, quantitative cost-benefit modelling before making this decision. This is very surprising to me – the least I would expect to see as part of a decision like this is an estimate of how it would affect the COVID exposure of attendees, and once such an estimate had been made I'm quite confident it should have been shared with the community.
- I'm also fairly confident that more transparency around the decision would have enabled others in the community to make, argue about, and refine these models, giving the community as a whole a much better idea of the relative costs and benefits of attending the conference.
- The above is a specific version of a general charge, which is that this decision, and other important COVID-related decisions made by the CEA events team, have been made in an untransparent and uncommunicative manner that (i) makes those decisions worse in expectation, and (ii) makes it difficult for the wider community to evaluate those decisions and make good decisions themselves about how to respond. Apart from those few people privileged to have close links with CEA or the Oxford office, most attendees didn't have access to the reasoning used to make the decision to expand. The arguments and data used to make that consequential decision should have been made available to the community, so that we could evaluate whether we found them convincing, challenge them if not, and update our beliefs and plans accordingly. This was not done.
- (This does not necessarily imply that the decision to expand or not expand should have been left up to to the community – this doesn't sound great to me for various reasons. Rather, I think CEA should have made its decision, then shared the in-depth reasoning for that decision with the community. I would make similar claims for all COVID-related events decisions by CEA.)
- [Low confidence] The decision to expand EA Global was inconsistent with CEA's previous claims about conference size.
- As mentioned above, EA Global conferences have been oversubscribed for many years now, in part because of a decision to intentionally reduce the size of the conferences: "The largest EA Global was about 1000 people in 2016, and we got feedback that it was too big and that it was easy to get lost in the shuffle. Our recent events have been between 500 - 650 people including speakers, volunteers, and staff."
- While I was never super confident in these arguments, they seem sufficient to make expanding the event ambiguously good even in the absence of COVID; it seems especially strange to reverse this policy at the time when the downside risk of doing so is drastically larger than it was in the past.
- My understanding now is that there are various reasons why CEA no longer believes these arguments – for example, the community is now larger, the number of high-quality applicants is higher, and software for connecting attendees and facilitating a good conference experience has improved.
- These seem like reasonable arguments to me – but I only learned about them by sharing a draft of this post with CEA! The decision to expand EAG London 2021 would have been a great opportunity to lay out those updates for the community, but this was not done.
- [Medium confidence] CEA undercompensated for the increase in risk.
- Based on my communications with CEA, I believe they did take some steps to increase the safety of the venue in response to the expansion, including communicating with the venue about increasing ventilation and setting up an additional marquee.
- These were good changes! The increase in ventilation in particular seems great to me.
- However, at the event itself I didn't get the sense that the event organisers were trying especially hard to reduce COVID risks from the event.
- The marquees, for example, were very well-sealed and robust, which is good for keeping atteendees warm but not for improving safety. One of the marquees had a large flap at the front which I and some others opened on Friday afternoon; however, this was closed on Friday night and never re-opened. Most windows at the event were also kept shut throughout the event, even when the weather was good. And as far as I know, no real effort was made to make sure attendees were getting tested – probably the most valuable intervention in terms of reducing risk without impairing the conference experience.
- (I'm not including masks here, because I think requiring masks would have made the experience of the event for many attendees much worse, which doesn't seem worth it to me.)
- (In the event, the weather was better than forecast and I was able to have a lot of meetings outside, circumventing some of these concerns. But that was sheer luck.)
- My loose impression from communicating with various people at or close to CEA is that they (or at least some of them) were operating under a model where the expected costs from COVID just weren't really something to worry too much about, relative to the other costs of operating such an event, and as such not worth putting too much effort or resources into mitigating. This explains many of my observations above, and might even be true – however, I don't think this was made at all clear to the community, and I think a number of attendees would have been surprised, and perhaps alarmed (not necessarily correctly!), to learn this was the case.
- [Fairly high confidence] Expanding after the fact was substantially worse than deciding on a larger conference from the beginning.
- Upon making the decision to expand the event, CEA offered to allow those who had signed up when it was due to have an attendance of 500 a free refund if they decided they didn't want to attend, as well as compensation for travel and accommodation. While this is all good, and I admit I'm not sure what else they should have done at that point given their other decisions, it's far from a great solution.
- People are generally much less likely to cancel something they no longer think is a good idea, than to book it in the first place. I think these sorts of psychological lock-in effects will have led to many less people deciding not to come than might, on deep reflection, have done so. Even if they did do so, overcoming these biases is much more costly in terms of time and effort than deciding not to attend ex ante would have been; in the words of Algorithms to Live By, it was computationally unkind.
- All these effects are exacerbated if they also have to cancel & seek compensation for travel & accommodation.
- Given this, being transparent about the reasoning underlying the decision was especially important.
- In addition, only deciding to expand the event a month in advance reduced CEA's (and other actors') ability to make changes to the event to compensate for the increased size. (Though as discussed above, I don't think they took full advantage of the opportunities they did have, so possibly this wouldn't have mattered too much.)
- For these and other reasons, I think changing the size of the event like this was significantly worse than deciding in advance on the size of the event and sticking to it.
In almost all things non-COVID, I believe that the CEA events team are some of the best events organisers in the world. They consistently put on events that far exceed the quality of any other conference series I have experience of, and EA Global London 2021 continued that trend. I attended the conference, and am confident it was good for the world for me to do so. But I wish the decision to expand the conference had been made more carefully, more transparently, and sooner.
Note/disclaimer: I recently started working on the Events Team (I do communications and impact assessment), and I shared this response with them before posting it (for edits and comments). Still, the thoughts here are mine.
The tl;dr:
I think you bring up some good points. I agree that we could have planned further in advance, been more transparent, and done more to reduce COVID risk at the event. I think these are important points, and I’ll discuss them below.
But in the end, I think it was right to increase the size of EAG: it roughly doubled the value produced by the event, and I think the increase in COVID risk was small compared to this benefit. I also think that there are some errors in your post. Finally, I want to note that our survey data suggests most attendees were happy with how we handled this situation.
So let’s get into it.
We should have put more effort into anticipating the quality of applications than we did (but we’re also not sure if we could have done that much better)
One of the claims that you make in this vein is:
This is true. Expanding capacity so late into the whole process surprised our attendees and speakers with last-minute changes and overwhelmed the team, impeding our ability to perfect various elements of the conference. You accurately note that “only deciding to expand the event a month in advance reduced CEA's (and other actors') ability to make changes to the event to compensate for the increased size.”
The key question, though, is whether we could have realized that we would need to make this change sooner. So on to your other point:
I think this is somewhat true, although I also think hindsight is particularly strong here. There were a number of difficult-to-predict factors at play. For instance, we provided more funding and support (for travel and accommodation) than we had ever before, which we guessed would increase attendance. But on the other hand, we thought that COVID caution might reduce the number of applications. I don’t think we had great data on how these factors would balance out.
Finally, just to clarify; the major surprise was not the number of applications, but rather their quality, which became clear after applications opened in September. In retrospect, we wish we had opened applications sooner.
Still, we should have spent more time trying to predict the number of great applications we’d get, and we generally could have been more careful with that process.
So I agree that in the future we should do better modeling, use forecasting for our planning, and be more aware of broad changes to the community. We’ll try to make explicit plans for being better at this in the future, and we welcome advice on how to do this.
It would have been better if we were more transparent about our reasoning
“The decision to expand the event was underjustified and untransparent.”
I agree.
To be clear, though, we were not trying to be untransparent: we were just busy with other things and didn’t prioritize this. At that point, it seemed that making sure that the schedule would make sense, that volunteers would know what to do, or that registration would happen was more urgent than writing up and publishing our thought process.
But I agree that by not doing this, our decisions may have seemed mysterious and underjustified. We didn’t explain what had changed since our 2019 posts, and this may not have been the right choice.
We made mistakes when attempting to make EA Global as safe as possible for attendees.
“CEA undercompensated for the increase in risk.”
Here are some things we did to make the event safer:
We also offered refunds (including for travel/accommodation) to everyone who signed up before we expanded the event, to allow people who weren’t comfortable with the larger event size to change their plans.
We considered some further measures, like checking that everyone completed tests every day. After assessing the risks and consulting with our advisory board, we decided that the benefits of verifying these (rather than trusting people to carry out tests as we asked) didn’t justify the significant costs.
But we made some errors here. For instance, we asked the venue to keep windows open, but they didn’t do this as much as we wanted them to. I think we should have pushed harder during the event for them to do what we had requested.
A note about how COVID risk relates to the decision to expand the conference. According to our model of COVID risk, the key factor in risk of infection is how densely packed attendees are. As we doubled the capacity, we also approximately doubled the size of the space available for lunch (and made sure that attendees actually spread out into this space) and other sessions. We did this by reserving a second venue (a good chunk of the Barbican) and adding the marquees for one-on-ones. So I think that although the risk increased a bit, it didn’t increase by that much (substantially less than by 100%).
The decision itself
It’s September and we’ve realized that the quality of applications we’re getting is far higher than we expected. We’re faced with the decision described above: do we grit our teeth and stick to our original plan, or increase the capacity two-fold?
When I first heard the proposal to expand the conference size by around 100% (the first proposal was actually a bit smaller), I thought it was insane. I imagined the chaos during registration, the confused emails, and, indeed, the increased COVID risk. We started brainstorming ways that might help us expand capacity without increasing COVID risk, and came up with the marquees and the second venue. I still thought the event would end up worse.
But then during a conversation with Amy, I realized that I was thinking from the point of view of a single attendee— someone admitted in the first wave of applications to an event for 500 people, who would suddenly be introduced to a more confusing conference for 1000. I could viscerally feel the loss in quality.
What I wasn’t imagining or feeling was the situation of an attendee who could only attend if we expanded capacity. They would be stripped of the chance to solicit feedback on their new animal welfare initiative, recruit candidates for their growing AI safety startup, get feedback on their big career decision, connect with potential funders or collaborators, or get a better understanding of the most pressing issues in the Global Health and Wellbeing space. This lost impact was invisible to my System One, but it would nevertheless be a massive amount of lost impact.
In other words, I realized that even if the conference were a bit worse for those who had already registered, it might roughly double the total value, by helping more people to attend.
In order to fulfill our obligations to the people who already registered, we offered full refunds (including travel expenses) to anyone who did not want to attend the larger event. (Although I acknowledge that psychological lock-in effects are real.)
I think that we managed the increase in capacity well, and we didn’t significantly worsen the experience relative to a 500 person conference: logistics were smooth, COVID risk wasn’t much higher, and they got access to nearly 500 extra people. In fact, due to the increased value from access to the extra attendees, I think the change might have been net positive for the initial 500 attendees (but we’re not sure overall).
In hindsight, I think the decision was correct.
Survey data
Here is some data from the feedback survey we sent to in-person attendees to back this up.
We asked attendees whether they wanted fewer or more attendees (a multi-part question introduced as “What changes should we make for future conferences?” where the options were "N/A" or "No strong opinion", Fewer, Keep the same, More).
Here is the result (587 responses):
We also asked “How satisfied were you with pre-conference communication?,” Of 596 responses, here are the results.
We asked “How satisfied were you with our COVID policy?” Results (593 responses):
[EDIT: Note that we plan to follow up with people who applied and were accepted but could not attend or cancelled their registration, and their responses to the above question may be different from those of the people who did come to EA Global. I’m guessing that in aggregate, the effect will be small due to the numbers (most people who were accepted did attend), but I will also consider those cases separately.]
We asked “How satisfied were you with COVID safety at EA Global in practice?” 595 responses. Result:
The average response for “How likely is it that you would recommend EA Global to a friend or colleague with similar interests to your own?” (on a scale from 0-10) is 9.11. For reference, this was 8.2 for EA Global: San Francisco 2019, 8.5 for EA Global: London 2019, and 7.8 for EA Global: Virtual in 2020.
I don’t add all of this here to claim that we did everything perfectly. I don’t think we did, and we plan to interview attendees with different perspectives to figure out how exactly we should improve in the future, both with respect to COVID and with respect to other aspects of running conferences.
But I do want to say that we took the decision to expand the conference (as well as the COVID implications of this decision) seriously, and I think it was the right decision.
In any case, I want to thank you for posting this on the Forum. I’m sorry for how these changes affected you, and I think it’s good for these criticisms to be discussed publicly.
I think some of the disconnect/confusion in the follow-on discussion here is being caused by my failing to update from a model of "CEA made a difficult COVID cost/benefit calculation" to one of "CEA didn't think COVID was a major concern" sufficiently quickly/completely. (Most of my post was written under the former model.)
I need to think more about that before I try to clarify my position here, because I'm not 100% sure what it is.