I am currently researching forecasting and causal inference. I am always excited to join forces to tackle important problems of any discipline or kind! Do not hesitate to reach out to me.
Do you think setting a day to talk with teams one-on-one before the event would help mitigate some of the randomness associated with team formation and get a better flavour of commitment? It will certainly give participants a better flavour of commitment. I think that this will lead to more early dropouts, so I have no idea whether this will be a sensible policy. I also suspect most will deem it unnecessary and overly paternalistic. Teams that take it seriously will meet up themselves.
One thing I'm piloting to improve follow up is setting a day to talk with teams about report progression, say a week after an event. Do you think that would help? Yes, definitely! One week is pretty short, but I think it would create a deadline (you have to have something baked together and prepared) that was lacking.
Also: I think that two attempts at replication games is not a good sample size (especially if the success and fun is not normally distributed), so I will attend another one. See you around!
There is one, and I think is the main flaw of the presented argument: A lot of problems are only considered pressing because of our expectations regarding the future. Improving the latter is not "solving anything", but simply clearing up priorities.
As someone who has dedicated their PhD to researching forecasting, I think this article raises an important point—but states it too bluntly, which ultimately muddles its central argument.
First, some context. Broadly speaking, all decisions are made on the basis of expectations about the future. It follows that anything which shifts those expectations can affect decisions in meaningful ways. This is easy to overlook, but it matters for how we evaluate forecasting interventions.
Furthermore, it is not easy to observe the effects of research. Consider the work of the Forecasting Research Institute. Much of what FRI does is research, and most of the benefits of research accrue in the future and are not easily traced back to any single input. The same difficulty applies more generally: since we cannot observe the counterfactual world in which a forecasting intervention was never made, measuring its effect is genuinely hard. How many decisions were improved? We may never know with precision.
That said, I do think the author raises a valid concern. We should be highly skeptical of the large investments of time and money by philanthropic actors and others in this space. Since many people find forecasting inherently enjoyable, there is a real risk that enthusiasm is doing some of the work that evidence should be doing—the elephant in the brain. I share that worry.
Where I part ways with the author is on the framing. The central argument of this piece should be: "We should rigorously review EA funding related to forecasting." That is a claim I agree with strongly. But the article instead reaches for a bolder headline—"Forecasting is overrated"—and there I disagree entirely. Most people in the world have never heard of structured forecasting, even though it could help a substantial number of them make better decisions in both business and personal life. Forecasting is not overrated. If anything, it is profoundly underrated.
My suggestion would be to get peers with actual topic-relevant knowledge to review funding proposals.
I am not sure whether this is currently done, but it is standard practice in governmental science funding.
I understand that it is the funders decision to collect peer reports, but I guess it is an easy way to spot projects that are targeted towards funders, not impact.
Hi Robert, thanks for answering. I would like to add two more points to what you already said:
1. To get more bang for relatively little organising, why not get just a few air filters for those rooms which are most crowded? 100+ air filters seem indeed like a lot (I calculated with 12).
2. It would perhaps be easier to get data on the health benefit related to reducing indoor air pollution. There are plenty of studies on this, I will try to find a good one. This alone may be sufficient to justify the expense. Reducing respiratory diseases would come on top.
I would like to see this discussion re-ignited. I got sick at EAGxBerlin2024, ironically missing a talk on „The economic value of reducing indoor infections“, which argues that UVC-lamps are a very cost-effective measure for indoor gatherings (https://www.d-fine.com/en/news/reducing-indoor-infections/).
I understand that it is difficult to estimate the effect of indoor air quality improvements to EA events (which include reduced air pollution AND reduced pathogen load), because these events may be different from what has been studied in the past, e.g. there already is existing ventilation, some people wear masks, many are young etc. I still encourage people with more expertise in the area to try to make an estimate (@Dawn Drescher - I could not reach your weblink).
I firmly believe that investing in air purification at EAG(x) is worthwhile, as the associated costs are relatively minor, and Eli_Nathan's arguments against this, as the former organizer, lack validity.
I list them below:
not sure there’s a way to rent air filtration equipment
Response: There is (now). The cost of renting 20 industrial-grade air purifiers for one weekend seems to be roughly 2500 pounds. Given the number of attendees at EAG, air purification is certainly cheaper than the shirts. (I just picked the first google response to base my estimate on: https://www.cas-hire.co.uk/air-cleaners/healthcare-bacterial-and-odour-control/)
quite logistically challenging
Response: How? As I understand it, air filter are directly shipped to the destination, and then shipped back afterwards. My volunteering experience tells me that organizers worry about stuff that I would consider of lesser importance, such as getting the number of toilets on descriptions exactly right. Renting a ton of furniture (as is the case for EAGxBerlin) surely is as logistically challenging, if not more.
there are also trade-offs re noise pollution
Response: What are these? Modern air filters tend to be relatively silent. In my experience, they are barely noticeable, even without anyone talking. Never has a guest at my house noticed the sound of my air filter (and the noise level is much lower than at EAG).
at the end of the day there are other marginal improvements to the conference I’m more excited about making
Response: Since when does it make sense to implement only the „best“ improvements? Is it not sufficient to robustly improve the event? I understand that the organizers do not want unnecessary additional hassle with this, but it does not seem to be more hassle than most other things at such events.
@Gordon Seidoh Worley also suggested that we bring our home equipment. Whilst I do not think that many are able and willing to do this, my air filter is lightweight and I would have been willing to bring it. If a small share (say 3%) of participants brought their equipment this could suffice as well.
I am tagging an event organizer here just so that this comment gets read by someone, @RobertHarling - the honor is yours.
I was surprised to see the comments on this post, which mostly provide arguments in favor of pursuing technological progress, even if this might lead to a higher risk of catastrophes.
I would like to chip in the following:
Preferences regarding the human condition are largely irrelevant for technological progress in the areas that you mention. Technological progress is driven by a large number of individuals that seek prestige and money. There is simply consumer demand for AI and technologies which may alter the human condition. Thus, technological progress happens, irrespective of whether this is considered good or bad.
Further reading:
The philosophical debate you are referring to is sometimes discussed as the scenario "1972", e.g. in Max Tegmarks "Life 3.0". He also provides reasons to believe that this scenario is not satisfying, given better alternatives.
Thanks Ren for this in-depth article. This is pure gold! Btw: I happened to read something related a couple of days ago: why-you-should-publish-your-research-in-academic-fashion. Maybe you should ask the author to link to your post?
Also: You have written "paper" instead of "journal" on the first line of your subsection Open access mega journals.
Hi Derek,
thanks for your kind reply.
Do you think setting a day to talk with teams one-on-one before the event would help mitigate some of the randomness associated with team formation and get a better flavour of commitment? It will certainly give participants a better flavour of commitment. I think that this will lead to more early dropouts, so I have no idea whether this will be a sensible policy. I also suspect most will deem it unnecessary and overly paternalistic. Teams that take it seriously will meet up themselves.
One thing I'm piloting to improve follow up is setting a day to talk with teams about report progression, say a week after an event. Do you think that would help? Yes, definitely! One week is pretty short, but I think it would create a deadline (you have to have something baked together and prepared) that was lacking.
Also: I think that two attempts at replication games is not a good sample size (especially if the success and fun is not normally distributed), so I will attend another one. See you around!
There is one, and I think is the main flaw of the presented argument: A lot of problems are only considered pressing because of our expectations regarding the future. Improving the latter is not "solving anything", but simply clearing up priorities.
As someone who has dedicated their PhD to researching forecasting, I think this article raises an important point—but states it too bluntly, which ultimately muddles its central argument.
First, some context. Broadly speaking, all decisions are made on the basis of expectations about the future. It follows that anything which shifts those expectations can affect decisions in meaningful ways. This is easy to overlook, but it matters for how we evaluate forecasting interventions.
Furthermore, it is not easy to observe the effects of research. Consider the work of the Forecasting Research Institute. Much of what FRI does is research, and most of the benefits of research accrue in the future and are not easily traced back to any single input. The same difficulty applies more generally: since we cannot observe the counterfactual world in which a forecasting intervention was never made, measuring its effect is genuinely hard. How many decisions were improved? We may never know with precision.
That said, I do think the author raises a valid concern. We should be highly skeptical of the large investments of time and money by philanthropic actors and others in this space. Since many people find forecasting inherently enjoyable, there is a real risk that enthusiasm is doing some of the work that evidence should be doing—the elephant in the brain. I share that worry.
Where I part ways with the author is on the framing. The central argument of this piece should be: "We should rigorously review EA funding related to forecasting." That is a claim I agree with strongly. But the article instead reaches for a bolder headline—"Forecasting is overrated"—and there I disagree entirely. Most people in the world have never heard of structured forecasting, even though it could help a substantial number of them make better decisions in both business and personal life. Forecasting is not overrated. If anything, it is profoundly underrated.
My suggestion would be to get peers with actual topic-relevant knowledge to review funding proposals.
I am not sure whether this is currently done, but it is standard practice in governmental science funding.
I understand that it is the funders decision to collect peer reports, but I guess it is an easy way to spot projects that are targeted towards funders, not impact.
Hi Robert, thanks for answering. I would like to add two more points to what you already said:
1. To get more bang for relatively little organising, why not get just a few air filters for those rooms which are most crowded? 100+ air filters seem indeed like a lot (I calculated with 12).
2. It would perhaps be easier to get data on the health benefit related to reducing indoor air pollution. There are plenty of studies on this, I will try to find a good one. This alone may be sufficient to justify the expense. Reducing respiratory diseases would come on top.
I would like to see this discussion re-ignited. I got sick at EAGxBerlin2024, ironically missing a talk on „The economic value of reducing indoor infections“, which argues that UVC-lamps are a very cost-effective measure for indoor gatherings (https://www.d-fine.com/en/news/reducing-indoor-infections/).
I understand that it is difficult to estimate the effect of indoor air quality improvements to EA events (which include reduced air pollution AND reduced pathogen load), because these events may be different from what has been studied in the past, e.g. there already is existing ventilation, some people wear masks, many are young etc. I still encourage people with more expertise in the area to try to make an estimate (@Dawn Drescher - I could not reach your weblink).
I firmly believe that investing in air purification at EAG(x) is worthwhile, as the associated costs are relatively minor, and Eli_Nathan's arguments against this, as the former organizer, lack validity.
I list them below:
Response: There is (now). The cost of renting 20 industrial-grade air purifiers for one weekend seems to be roughly 2500 pounds. Given the number of attendees at EAG, air purification is certainly cheaper than the shirts. (I just picked the first google response to base my estimate on: https://www.cas-hire.co.uk/air-cleaners/healthcare-bacterial-and-odour-control/)
quite logistically challenging
Response: How? As I understand it, air filter are directly shipped to the destination, and then shipped back afterwards. My volunteering experience tells me that organizers worry about stuff that I would consider of lesser importance, such as getting the number of toilets on descriptions exactly right. Renting a ton of furniture (as is the case for EAGxBerlin) surely is as logistically challenging, if not more.
there are also trade-offs re noise pollution
Response: What are these? Modern air filters tend to be relatively silent. In my experience, they are barely noticeable, even without anyone talking. Never has a guest at my house noticed the sound of my air filter (and the noise level is much lower than at EAG).
at the end of the day there are other marginal improvements to the conference I’m more excited about making
Response: Since when does it make sense to implement only the „best“ improvements? Is it not sufficient to robustly improve the event? I understand that the organizers do not want unnecessary additional hassle with this, but it does not seem to be more hassle than most other things at such events.
@Gordon Seidoh Worley also suggested that we bring our home equipment. Whilst I do not think that many are able and willing to do this, my air filter is lightweight and I would have been willing to bring it. If a small share (say 3%) of participants brought their equipment this could suffice as well.
I am tagging an event organizer here just so that this comment gets read by someone, @RobertHarling - the honor is yours.
I was surprised to see the comments on this post, which mostly provide arguments in favor of pursuing technological progress, even if this might lead to a higher risk of catastrophes.
I would like to chip in the following:
Preferences regarding the human condition are largely irrelevant for technological progress in the areas that you mention. Technological progress is driven by a large number of individuals that seek prestige and money. There is simply consumer demand for AI and technologies which may alter the human condition. Thus, technological progress happens, irrespective of whether this is considered good or bad.
Further reading:
The philosophical debate you are referring to is sometimes discussed as the scenario "1972", e.g. in Max Tegmarks "Life 3.0". He also provides reasons to believe that this scenario is not satisfying, given better alternatives.
Thank you so much! This helps me make a decision.
Thanks Ren for this in-depth article. This is pure gold! Btw: I happened to read something related a couple of days ago: why-you-should-publish-your-research-in-academic-fashion. Maybe you should ask the author to link to your post?
Also: You have written "paper" instead of "journal" on the first line of your subsection Open access mega journals.
Thanks so much for the review! I would like to add that there is some evidence that simple acupressure mats help alleviate low back pain.