Thanks. A quick, non-exhaustive list:
There's another point I don't quite know how to put but I'll give it a go.
Despite the comments above about having many ideas and getting feedback early about one's projects - which both point to having and abandoning ideas quickly - there's another sense in which actually what one needs is an ability to stick to things. And the good taste to be able to evaluate when to try something else and when to keep going. (This is less about specific projects and more about larger shifts like whether to stay in academia/a certain line of work at all.)
I feel like some...
I've stayed at a (non-EA) professional contact's house before when they'd invited me to give a talk and later very apologetically realized they didn't have the budget for a hotel. They likely felt obliged to offer; I felt like it would be awkward to decline. We were both at pains to be extremely, exceedingly, painstakingly polite given the circumstances and turn the formality up a notch.
I agree the org should have paid for a hotel, I'm only mentioning this because if baseline formality is a 5, I would think it would be more normal to kick it up to a 10 under the circumstances. It makes this situation all the more bizarre.
I would really like to read a summary of this book. The reviews posted here (edit: in the original post) do not actually give much insight as to the contents. I'm hoping someone will post a detailed summary on the forum (and, as EAs love self-criticism, fully expect someone will!).
I'm not going to deal with the topic of the post, but there's another reason to not post under a burner account if it can be avoided that I haven't seen mentioned, which this post indirectly highlights.
When people post under burner accounts, it makes it harder to be confident in the information that the posts contain, because there is ambiguity and it could be the same person repeatedly posting. To give one example (not the only one), if you see X number of burner accounts posting "I observe Y", then that could mean anywhere from 1 to X observations of Y, ...
Here's an example of a past case where a troll (who also trolled other online communities) made up multiple sock-puppet accounts, and assorted lies about sources for various arguments trashing AI safety, e.g. claiming to have been at events they were not and heard bad things, inventing nonexistent experts who supposedly rejected various claims, creating fake testimonials of badness, smearing people who discovered the deception, etc.
One thing I'd like to quickly flag on the topic of this comment: using multiple accounts to express the same opinion (e.g. to create the illusion of multiple independent accounts on this topic) is a (pretty serious) norm violation. You can find the full official norms for using multiple accounts here.
This doesn't mean that e.g. if you posted something critical of current work on forecasting at some point in your life, you can't now use an anonymous account to write a detailed criticism of a forecasting-focused organization. But e.g. commenting on the...
Thanks for mentioning the Social Science Prediction Platform! We had some interest from other sciences as well.
With collaborators, we outlined some other reasons to forecast research results here: https://www.science.org/doi/10.1126/science.aaz1704. In short, forecasts can help to evaluate the novelty of a result (a double-edged sword: very unexpected results are more likely to be suspect), mitigate publication bias against null results / provide an alternative null, and over time help to improve the accuracy of forecasting. There are other reasons, as wel...
As a small note, we might get more precise estimates of the effects of a program by predicting magnitudes rather than whether something will replicate (which is what we're doing with the Social Science Prediction Platform). That said, I think a lot of work needs to be done before we can have trust in predictions, and there will always be a gap between how comfortable we are extrapolating to other things we could study vs. "unquantifiable" interventions.
(There's an analogy to external validity here, where you can do more if you can assu...
Great comment. I don't think anyone, myself included, would say the means are not the same and therefore everything is terrible. In the podcast, you can see my reluctance to that when Rob is trying to get me to give one number that will easily summarize how much results in one context will extrapolate to another, and I just don't want to play ball (which is not at all to criticize!). The number I tend to focus on these days (tau squared) is not one that is easily interpretable in that way - instead, it's a measure of the unexplained variation in results - ...
This video might also add to the discussion - the closing panel at CSAE this year was largely on methodology, moderated by Hilary Greaves (head of the new Global Priorities Institute at Oxford), with Michael Kremer, Justin Sandefur, Joseph Ssentongo, and myself. Some of the comments from the other panellists still stick with me today.
https://ox.cloud.panopto.eu/Panopto/Pages/Viewer.aspx?id=ec3f076c-9c71-4462-9b84-a8a100f5a44c
And when groups do work on these issues there is a tendency towards infighting.
Some things that could help:
I agree that it would be important to weigh the costs and benefits - I don't think it's exclusively an issue with RCTs, though.
One thing that could help in doing this calculus is a better understanding of when our non-study-informed beliefs are likely to be accurate.
I know at least some researchers are working in this area - Stefano DellaVigna and Devin Pope are looking to follow up their excellent papers on predictions with another one looking at how well people predict results based on differences in context, and Aidan Coville and I also have some work in this area using impact evaluations in development and predictions gathered from policymakers, practitioners, and researchers.
There's another point I don't quite know how to put but I'll give it a go.
Despite the comments above about having many ideas and getting feedback early about one's projects - which both point to having and abandoning ideas quickly - there's another sense in which actually what one needs is an ability to stick to things. And the good taste to be able to evaluate when to try something else and when to keep going. (This is less about specific projects and more about larger shifts like whether to stay in academia/a certain line of work at all.)
I feel like some... (read more)