1 min read 8

5

Are you conducting some kind of experiment, or planning to? Well, I don't know about it yet! So when the stunning results come out, I can say "hey, maybe you just got lucky. If your experiment had returned a negative result then we wouldn't have heard anything about it".

This is called publication bias.

So I'm launching the anti publication bias registry, a place to record EA-related experiments before you know what the results are going to be.

It's on a wiki, which allows you to change the description afterwards - this has good points and bad points. A good point is that it allows you to change your experimental design as you go along, in order to best fit the world. A bad point is that this can introduce bias. But remember, we do have the edit history so blatant cheating will be hard (such as completely changing an experiment, or deleting it). More subtle cheating is also discouraged, such as adding extra statistical tests because the data seem to be pointing that way.

You can also register experiments by commenting on this post - someone will probably copy it to the wiki eventually anyway.

Be as detailed as you can be bothered to be in your experiment descriptions. This will hopefully encourage others to follow your example and be careful in how they set up their experiments.

A second purpose of this is to introduce social commitment towards actually completing and writing up experiments once they've been suggested.

Have fun doing science!

5

0
0

Reactions

0
0
Comments8


Sorted by Click to highlight new comments since:

Thanks, in general pre-registration is a great tool to help have appropriate confidence in results.

I may disagree with this slightly:

More subtle cheating is also discouraged, such as adding extra statistical tests because the data seem to be pointing that way.

There are some forms of that which are pure p-value hacking, and should definitely be discouraged. But experiments have a couple of purposes, only one of which is hypothesis testing. The other is exploration, which can help with hypothesis generation.

It's absolutely legitimate to notice an odd correlation between variables you didn't plan in advance to test for correlation and form a hypothesis about this. It may be helpful to use a statistical test to see how suggestive the data is. What you shouldn't do is confuse the strength of evidence for such hypotheses with ones which you set out to explore.

I agree. Thanks for clarifying.

If this turns out to something people find useful, it might also be useful to have people who watch the wiki and provide feedback/advice on the proposed study designs, or who can help people who are less familiar with study design and statistics to produce something useful. This provides an additional service along with the preregistration, so it isn't just an extra onerous task. (I'd be willing to do this if it seems useful).

I'm somewhat doubtful that this experiment registry will attract a lot of use, but +1 for setting it up to try it out.

There's a lot about this idea that I agree with. It seems important to get effective altruists (and everyone) to share more information about what they're trying to achieve, whether it worked or didn't, and why. In particular, a lot of us are trying radical things like applying for finance jobs or starting companies, which have high variance, making it hard to infer whether these are good decisions.

I think this idea can grow a lot. It would be good to have everyone we know pooling together information about when they started a company, for example, so that we could infer whether it's better to start more companies or work for somebody else. In the long run, it would be ideal situation would be to have thousands of people making their decision in similar ways to you, and to have a recommender system that can give you suggestions by giving extra weighting to the experiences of people similar to you.

However, I do have a bunch of feedback and questions:

1 Is a registry the best way of sharing information about what activities are working well and which aren't?

2 Is there some reason to focus just on explicit experiments rather than lots of activities with uncertain payoff? Are there enough explicit experiments to support this project? Don't we want to reduce reporting bias in general, rather than just publication bias? i.e. we want people to report the successes and failures of activities with uncertain payoff, rather than just explicit experiments. So it might be better to call it a Reporting Bias Registry.

3 On second thoughts, shouldn't it be named by what it's trying to achieve rather than what it's trying to avoid? e.g. the 'charity project performance registry', or 'social impact registry', or something like that.

4 I think it's important to be clear about what exactly you want people to do. At the top or bottom of the page, you could write in bold that what you want is for people to write information about their projects on the wiki.

5 It's not clear that a wiki is the best way to implement this. Few people don't use the wiki, and some of the important experiments that people run might be private? Perhaps it would be better to make a Google form, and to assure people that any information that is sufficiently specific to identify them or their project will not be disclosed? Or perhaps privacy would only work if there are at least dozens of projects. It's at least worth thinking about.

6 Is this substantially different from the EA Survey? Is it substantially different from the EA Profiles? Is it substantially different from GWWC's donation registry? CFAR's alumni community? 80k's alumni group? Can it be integrated with any of these things?

  1. Maybe we should also get everyone to report their progress on the projects that we already know about. This is kind-of different, because the reports will be somewhat biased but it still seems worthwhile.

So I would be interested in how you would respond to some of these challenges and how we could plan around them in order to make the project more likely to be the big success that the idea deserves.

There are plenty of valuable thoughts here. I also like Giles' idea and think it's worth giving a go.

A few specific comments:

It's not clear that a wiki is the best way to implement this.

I think a wiki's fine for now, and pretty simple. I don't think there's yet reason to worry about privacy. In general I think Giles' setup is a decent minimum viable product.

Maybe we should also get everyone to report their progress on the projects that we already know about. This is kind-of different, because the reports will be somewhat biased but it still seems worthwhile.

I believe .impact was set up partly for this purpose, and its projects page is a pretty good place to report this sort of thing.

I have a bunch of experiments I ran for a Master's Thesis related to the use of neural networks for object recognition, that ended up getting published in a couple conference papers. Given that any A.I. research has the potential to contribute to Friendly A.I., would those have counted or are they too distant from E.A.?

I also have an experiment that's current status is failed, a Neural Network Earthquake Predictor, but which I'm considering resurrecting in the near future by applying different and newer methods. How would I go about incorporating such an experiment into this registry, given that it technically has a tentative result, but the result isn't final yet?

Just an update. I decided to make a go of adding the experiment to the Registry. Hopefully what I added is acceptable. If not, let me know what I should change.

I had a look at the current entries, and it's a bit unclear to me. For the facebook welcome, I don't understand the methodology. Are you comparing different standard greetings against each other? If so, how do you decide which greeting gets used for which person?

In general the idea with pre-registration should be to state up-front exactly what questions you are trying to answer. This is a non-negligible amount of work involved in doing this properly; this is the cost of pre-registration (and why it isn't a total no-brainer, particularly early on in a field when exploration and hypothesis generation is more important than hypothesis testing). Of course bringing down the cost helps make it worthwhile earlier.

Curated and popular this week
Sam Anschell
 ·  · 6m read
 · 
*Disclaimer* I am writing this post in a personal capacity; the opinions I express are my own and do not represent my employer. I think that more people and orgs (especially nonprofits) should consider negotiating the cost of sizable expenses. In my experience, there is usually nothing to lose by respectfully asking to pay less, and doing so can sometimes save thousands or tens of thousands of dollars per hour. This is because negotiating doesn’t take very much time[1], savings can persist across multiple years, and counterparties can be surprisingly generous with discounts. Here are a few examples of expenses that may be negotiable: For organizations * Software or news subscriptions * Of 35 corporate software and news providers I’ve negotiated with, 30 have been willing to provide discounts. These discounts range from 10% to 80%, with an average of around 40%. * Leases * A friend was able to negotiate a 22% reduction in the price per square foot on a corporate lease and secured a couple months of free rent. This led to >$480,000 in savings for their nonprofit. Other negotiable parameters include: * Square footage counted towards rent costs * Lease length * A tenant improvement allowance * Certain physical goods (e.g., smart TVs) * Buying in bulk can be a great lever for negotiating smaller items like covid tests, and can reduce costs by 50% or more. * Event/retreat venues (both venue price and smaller items like food and AV) * Hotel blocks * A quick email with the rates of comparable but more affordable hotel blocks can often save ~10%. * Professional service contracts with large for-profit firms (e.g., IT contracts, office internet coverage) * Insurance premiums (though I am less confident that this is negotiable) For many products and services, a nonprofit can qualify for a discount simply by providing their IRS determination letter or getting verified on platforms like TechSoup. In my experience, most vendors and companies
 ·  · 4m read
 · 
Forethought[1] is a new AI macrostrategy research group cofounded by Max Dalton, Will MacAskill, Tom Davidson, and Amrit Sidhu-Brar. We are trying to figure out how to navigate the (potentially rapid) transition to a world with superintelligent AI systems. We aim to tackle the most important questions we can find, unrestricted by the current Overton window. More details on our website. Why we exist We think that AGI might come soon (say, modal timelines to mostly-automated AI R&D in the next 2-8 years), and might significantly accelerate technological progress, leading to many different challenges. We don’t yet have a good understanding of what this change might look like or how to navigate it. Society is not prepared. Moreover, we want the world to not just avoid catastrophe: we want to reach a really great future. We think about what this might be like (incorporating moral uncertainty), and what we can do, now, to build towards a good future. Like all projects, this started out with a plethora of Google docs. We ran a series of seminars to explore the ideas further, and that cascaded into an organization. This area of work feels to us like the early days of EA: we’re exploring unusual, neglected ideas, and finding research progress surprisingly tractable. And while we start out with (literally) galaxy-brained schemes, they often ground out into fairly specific and concrete ideas about what should happen next. Of course, we’re bringing principles like scope sensitivity, impartiality, etc to our thinking, and we think that these issues urgently need more morally dedicated and thoughtful people working on them. Research Research agendas We are currently pursuing the following perspectives: * Preparing for the intelligence explosion: If AI drives explosive growth there will be an enormous number of challenges we have to face. In addition to misalignment risk and biorisk, this potentially includes: how to govern the development of new weapons of mass destr
jackva
 ·  · 3m read
 · 
 [Edits on March 10th for clarity, two sub-sections added] Watching what is happening in the world -- with lots of renegotiation of institutional norms within Western democracies and a parallel fracturing of the post-WW2 institutional order -- I do think we, as a community, should more seriously question our priors on the relative value of surgical/targeted and broad system-level interventions. Speaking somewhat roughly, with EA as a movement coming of age in an era where democratic institutions and the rule-based international order were not fundamentally questioned, it seems easy to underestimate how much the world is currently changing and how much riskier a world of stronger institutional and democratic backsliding and weakened international norms might be. Of course, working on these issues might be intractable and possibly there's nothing highly effective for EAs to do on the margin given much attention to these issues from society at large. So, I am not here to confidently state we should be working on these issues more. But I do think in a situation of more downside risk with regards to broad system-level changes and significantly more fluidity, it seems at least worth rigorously asking whether we should shift more attention to work that is less surgical (working on specific risks) and more systemic (working on institutional quality, indirect risk factors, etc.). While there have been many posts along those lines over the past months and there are of course some EA organizations working on these issues, it stil appears like a niche focus in the community and none of the major EA and EA-adjacent orgs (including the one I work for, though I am writing this in a personal capacity) seem to have taken it up as a serious focus and I worry it might be due to baked-in assumptions about the relative value of such work that are outdated in a time where the importance of systemic work has changed in the face of greater threat and fluidity. When the world seems to