Effective Altruism Forum
EA Forum

Hide table of contents
Comment Permalink
Shakked Noy4
0
0

I think the IGM panel is great, and expanding it within economics or extending it to other fields would be great. Some quick thoughts:

It's impossible to objectively retrospectively evaluate panel responses, because all of the interesting questions are about causal effects of X on Y, where the answer doesn't become clear over time because the counterfactual is never observed. Maybe the best you could do is run a retrospective survey where you ask panel members (or other economists) whether they think the panel responses were right in hindsight.

I'm not surprised that Tyler Cowen has a negative view of many of the panel results; he's a libertarian and the panel is broadly left-leaning, as is the broader economics profession. I think a better critique is one that Cowen expresses, e.g., in his interview on Spencer Greenberg's podcast: that the views of economists seem to covary pretty closely with the political beliefs of their demographic group (highly educated urban dwellers), suggesting that their views aren't purely reflective of professional expertise.

I also don't think it's concerning that the IGM forum "excludes certain schools of economics, like the Austrian school." The economics profession doesn't really have "schools" anymore, and I don't think the "Austrian school" has ever had much mainstream credibility. The first and third bullet points from the 2016 paper are more concerning.

I don't think the IGM panel influences future research topics, as the questions are typically about topics already highly salient to academic economists. I don't know whether it influences public discourse or policymakers. The panel has high buy-in from academics because its members are all highly regarded in the profession.

Reply
3
MaxRa
Thanks, that all is useful feedback. Regarding the impossibility of resolving questions about causal effects: Take the example I gave in the screenshot. I would think that there are plausible observations in the coming years that make the causal claim "monopolists using their market power -> inflation" very unlikely, right? E.g. if the inflation decreases without observing changes in competitiveness or something like this. And yes, a retrospective survey might be the gold standard, though I suppose I'd feel somewhat comfortable trusting a careful economist to make a bunch of judgements by herself. And regarding buy-in, do you have an idea how the members themselves initially got interested in participating? I guess if you want to replicate the IGM panel, you'd need to have some well connected people on board fairly early. Maybe due to econs proximity to politics it's also very natural for them to want to speak out about policy issues.
3
Shakked Noy
I'm skeptical that there'll be a consensus 5 or 10 years from now on whether market power was a key contributor to 2021 inflation; there are just too many confounding factors blocking an inference like "inflation decreased without changes in competitiveness, so market power must not have been causing inflation." It's true that retrospective consensus sometimes emerges (for example, that financial deregulation was at least partly responsible for the Global Financial Crisis), but this is pretty rare. I'm not sure how the forum started. I assume a key factor in making the participants interested was that the panel was started by a reputable institution (Chicago Booth) that could guarantee a wide audience. I wouldn't guess that economists are more willing than academics from other fields to express views on policy-relevant issues; I suspect political scientists, sociologists, etc. would be similarly willing.
MaxRa3
0
0

Thanks, that makes sense!

Reply
See in context
by MaxRa
2 min read 11

53

Opportunities to take actionPolicyEconomicsImproving institutional decision-making
Frontpage

Surveying scientific experts seems like a promising way to gather informed opinions relevant for important decisions, and to improve common knowledge on important topics.

The IGM economic experts panel has been doing surveys for roughly 10 years, and I wonder how well both the surveyed consensus and the format of the platform hold up, as it might be a good idea to

  1. copy the general IGM platform to other scientific fields,
  2. offer ideas to improve the IGM panel, and more ambitiously to improve the economics consensus (e.g. by spotting biases), and
  3. use the survey results when they interact with topics relevant for EA.
Example for recent survey responses

Two ideas how to evaluate the survey's usefulness

  1. Look through IGM's survey questions and evaluate those which have something like a "correct" answer in retrospect. This requires somebody fairly informed about the surveyed issues.
  2. Interview someone from IGM, or a researcher knowledgeable about expert consensuses in economics or generally (suggestions?).

Two articles about the IGM survey

  • Tyler Cowen evaluating some IGM polls in 2015: "But quite seriously, my opinion of the professional consensus — on topics outside an individual’s research specialty — really has gone down as a result of these polls." (link)
  • discussion of biases in the economics elite, in the surveyed IGM questions and for which experts were selected (paper from 2016)
    • IGM asking softball questions apparently aiming at eliciting textbook answers
    • some schools of economics not being represented, e.g. the Austrian school
    • clustering of responses according to political partisanship and departmental affiliations
  • what has been the impact of the IGM survey?
    • did it influence policy makers? did it influence public discourse?
    • does it have an impact on the future research topics of surveyed researchers?
  • which fields are most promising candidates for a similar survey platform? are there already similar surveys in other fields?
  • what influences the buy-in that the IGM experts panel enjoys from academics?
  • what tradeoffs are involved in different formats of knowledge elicitation?

53

0
0

Reactions

0
0
Comments11
Sorted by
New & upvoted
Click to highlight new comments since:
AppliedDivinityStudies14
0
0

If you read the expert comments, very often they complain that the question is poorly phrased. It's typically about wording like "would greatly increase" where there's not even an attempt to define "greatly". So if you want to improve the panel or replicate it, that is my #1 recommendation.

...My #2 recommendation is to create a Metaculus market for every IGM question and see how it compares.

Reply
Larks8
0
0

Additionally, sometimes the question seems to ask about one specific cost or benefit of a policy, and respondents are unsure how to answer if they think that issue is unimportant but disagree/agree for other reasons.

Reply
Stephen Clare6
0
0

This does seem useful. At least one similar survey does exist for other fields: the TRIP survey for international relations scholars. I've found this somewhat useful for my research, though often the questions in IR seem less specific than the questions asked of economists.

Reply
RyanCarey5
0
0

There is also the PhilPapers Survey!

Reply
MichaelA🔸4
0
0

Thanks for this post.

Here's a copy of some quick notes I wrote earlier (and don't plan to take any next steps on), in case this is useful to anyone:

  • "I see results from the IGM economics expert panels cited somewhat often, and it seems like that's probably a really useful thing to have.
  • Idea 1: Does anyone know if there are similar things for other disciplines? 
    • E.g., historians, psychologists, international relations scholars?
    • And if there isn't, might it be highly valuable per unit effort to set that up?
    • Obviously lots of individual experts comment on individual things, and there'll sometimes be surveys or the like. But it seems like it could make sense for this sort of thing to be institutionalised in the way the IGM panels are, rather than being ad hoc.
  • Idea 2: Does anyone know if it's possible to submit questions to the IGM panel or other panels (if they exist), or to in some other way get them to respond to questions on topics of particular interest to EAs?
  • (If people think this sort of thing doesn't exist yet but might be worth thinking more about, I might write a quick EA Forum post/question post/shortform on this.)"

Replies from colleagues of mine included:

  • “In the US, the National Academy of Sciences, Engineering and Medicine does expert opinion in a more formal way for relevant issues e.g., https://www.nap.edu/catalog/25665/heritable-human-genome-editing”
  • “You hear a lot in the US from the US Army Corps of Engineers.”
  • “PhilPapers seems like a natural place to expand polls of philosophers.”
  • “I don't know about historians, but wouldn't it be cool if macrohistory/predictive history were more of a thing and they took surveys of their members?”
Reply
MaxRa2
0
0

Thanks, that's useful to me! E.g. I didn't consider trying to convince those surveying institutions to ask about particularly important topics they might not have on their radar, or maybe paying them for the service.

As you don't plan considering it further, did you make a guess how useful work on this seems? For fields that are particularly relevant for EAs (like epidemiology, AI, maybe international relations) it might be very valuable to take the initiative so that at least some share of the surveys will be informative for the most important issues.

Reply
MichaelA🔸4
0
0

I don't have a particularly informed/informative stance on how useful it'd be, unfortunately. (I guess I can at least say it'd be nice if someone spent a few hours thinking further about it, but that's true of many things.)

Reply
Shakked Noy4
0
0

I think the IGM panel is great, and expanding it within economics or extending it to other fields would be great. Some quick thoughts:

It's impossible to objectively retrospectively evaluate panel responses, because all of the interesting questions are about causal effects of X on Y, where the answer doesn't become clear over time because the counterfactual is never observed. Maybe the best you could do is run a retrospective survey where you ask panel members (or other economists) whether they think the panel responses were right in hindsight.

I'm not surprised that Tyler Cowen has a negative view of many of the panel results; he's a libertarian and the panel is broadly left-leaning, as is the broader economics profession. I think a better critique is one that Cowen expresses, e.g., in his interview on Spencer Greenberg's podcast: that the views of economists seem to covary pretty closely with the political beliefs of their demographic group (highly educated urban dwellers), suggesting that their views aren't purely reflective of professional expertise.

I also don't think it's concerning that the IGM forum "excludes certain schools of economics, like the Austrian school." The economics profession doesn't really have "schools" anymore, and I don't think the "Austrian school" has ever had much mainstream credibility. The first and third bullet points from the 2016 paper are more concerning.

I don't think the IGM panel influences future research topics, as the questions are typically about topics already highly salient to academic economists. I don't know whether it influences public discourse or policymakers. The panel has high buy-in from academics because its members are all highly regarded in the profession.

Reply
MaxRa3
0
0

Thanks, that all is useful feedback.

Regarding the impossibility of resolving questions about causal effects: Take the example I gave in the screenshot. I would think that there are plausible observations in the coming years that make the causal claim "monopolists using their market power -> inflation" very unlikely, right? E.g. if the inflation decreases without observing changes in competitiveness or something like this. And yes, a retrospective survey might be the gold standard, though I suppose I'd feel somewhat comfortable trusting a careful economist to make a bunch of judgements by herself.

And regarding buy-in, do you have an idea how the members themselves initially got interested in participating? I guess if you want to replicate the IGM panel, you'd need to have some well connected people on board fairly early. Maybe due to econs proximity to politics it's also very natural for them to want to speak out about policy issues.

Reply
Shakked Noy3
0
0

I'm skeptical that there'll be a consensus 5 or 10 years from now on whether market power was a key contributor to 2021 inflation; there are just too many confounding factors blocking an inference like "inflation decreased without changes in competitiveness, so market power must not have been causing inflation." It's true that retrospective consensus sometimes emerges (for example, that financial deregulation was at least partly responsible for the Global Financial Crisis), but this is pretty rare.

I'm not sure how the forum started. I assume a key factor in making the participants interested was that the panel was started by a reputable institution (Chicago Booth) that could guarantee a wide audience. I wouldn't guess that economists are more willing than academics from other fields to express views on policy-relevant issues; I suspect political scientists, sociologists, etc. would be similarly willing.

Reply
MaxRa3
0
0

Thanks, that makes sense!

Reply
More from MaxRa
View more
Curated and popular this week
Relevant opportunities
View more