Sequences
1
I'm the Founder and Co-director of The Unjournal; We organize and fund public journal-independent feedback, rating, and evaluation of hosted papers and dynamically-presented research projects. We will focus on work that is highly relevant to global priorities (especially in economics, social science, and impact evaluation). We will encourage better research by making it easier for researchers to get feedback and credible ratings on their work.
Previously I was a Senior Economist at Rethink Priorities, and before that n Economics lecturer/professor for 15 years.
I'm working to impact EA fundraising and marketing; see https://bit.ly/eamtt
And projects bridging EA, academia, and open science.. see bit.ly/eaprojects
My previous and ongoing research focuses on determinants and motivators of charitable giving (propensity, amounts, and 'to which cause?'), and drivers of/barriers to effective giving, as well as the impact of pro-social behavior and social preferences on market contexts.
Podcasts: "Found in the Struce" https://anchor.fm/david-reinstein
and the EA Forum podcast: https://anchor.fm/ea-forum-podcast (co-founder, regular reader)
Twitter: @givingtools
Thanks Bob. I agree this would be high value. I've been mainly thinking about human/LLM agreement, but it would be useful to know how consistent each model is, how sensiitive it is to the prompts, whether the models differ in systematic ways, etc.
Actionable takeaways from doing this? Is a 'more stable setup' (one that e.g., has fairly consistent rank orderings) better all else equal, and thus something we should work towards? Probably, as the alternative seems like just 'adding noise'.
Knowing the stability also may tells us how much effort to put into designing and building consensus around a 'reasonable prompt'; if the results are insensitive to this, we shouldn't waste too much time.
Working on implementing something now, at least a first step. I hope to report back on some legible measures of prompt stability (or lack thereof).
Project Idea: 'Cost to save a life' interactive calculator promotion
What about making and promoting a ‘how much does it cost to save a life’ quiz and calculator.
This could be adjustable/customizable (in my country, around the world, of an infant/child/adult, counting ‘value added life years’ etc.) … and trying to make it go viral (or at least bacterial) as in the ‘how rich am I’ calculator?
The case
While GiveWell has a page with a lot of tech details, but it’s not compelling or interactive in the way I suggest above, and I doubt they market it heavily.
GWWC probably doesn't have the design/engineering time for this (not to mention refining this for accuracy and communication). But if someone else (UX design, research support, IT) could do the legwork I think they might be very happy to host it.
It could also mesh well with academic-linked research so I may have some ‘Meta academic support ads’ funds that could work with this.
Tags/backlinks (~testing out this new feature)
@GiveWell @Giving What We Can
Projects I'd like to see
EA Projects I'd Like to See
Idea: Curated database of quick-win tangible, attributable projects
Help The Unjournal prioritize AI economics and governance research
(https://uj-prioritization-dashboard.netlify.app/ai-governance/)
The Unjournal (https://info.unjournal.org/) is prioritizing research on AI economics, governance, social and economic impacts, and risks from increasingly capable AI. (NB: We're not covering technical/CS/ML/AI-safety research.). We want to commission expert public evaluations and help synthesize, disseminate, and curate this work.
Much of this research seems 1) important/impactful/influential, (2) involves considerable 'firepower' from prominent researchers and practitioners, (3) not 'obviously true', and (4) time-sensitive: relevant for funding, policy, and research-steering now, but not after the "six months to six years" required for the traditional (economics) journal process
Our preliminary shortlist page (https://uj-prioritization-dashboard.netlify.app/ai-governance/) includes paper summaries, AI-generated audio walkthroughs, and an interface for giving feedback. We also invite suggestions fr other research (please check the broader list/interface to avoid duplication).
Especially looking for your input on:
Timing and next steps (our plan)
If it this works well, we hope to scale up and continue in the following months.
Alongside the evaluations, we want to help people find and use this research, and to help shape its direction. I hope the prioritization itself will be useful to students, researchers, practitioners, and other stakeholders. (NB: the "prioritization" here is mainly about "potential for impact and value for evaluation"; assessing the research credibility is for the evaluation part).
I wanted to share this quickly; planning a fuller post soon. (Feedback on this post itself is helpful too.)
Related: We’re also exploring sponsorship of evaluations/themes/questions (https://uj-prioritization-dashboard.netlify.app/sponsor/). Looking for feedback: Would this be useful, would you sponsor or recommend others to do so? Would sponsorship affect your trust in the evaluations (given the rules and safeguards mentioned in the link)?
And an (AI generated, at my prompting) comparison of the models here https://uj-ai-wealth-philanthropy-steelman.netlify.app/model-comparison/ ... I'll try to dig in more on this soon.
https://uj-ai-wealth-philanthropy-steelman.netlify.app/model-comparison/ comparing this to others' models and work (I'll try to dig in more on that later)
https://uj-ai-wealth-philanthropy-steelman.netlify.app/model-comparison/ comparing these (I'll try to dig in more on that later)
Proposing an improvement to otols like this -- suggestionh bounties w external judgment and/or 'enforcement' -- https://forum.effectivealtruism.org/posts/oxFncTiAaHBJCgBZP/anonymous-suggestion-boxes-with-a-committed-reward-pool-and?sharePopup=true
Also please see the model that I created, forking into a specific fork for development economics. Would be good to compare and potentially collaborate on these soon. Interesting to see what semi-independent modeling exercises come up with, how much they converge and diverge, etc.
https://forum.effectivealtruism.org/posts/RLWpq2oXPsunoqTEW/live-models-and-dashboards-of-anticipated-ai-wealth-going-to
https://uj-ai-wealth-philanthropy-steelman.netlify.app/global-development/
https://uj-ai-wealth-philanthropy-steelman.netlify.app/
Also please see the model that I created, forking into a specific fork for development economics. Would be good to compare and potentially collaborate on these soon. Interesting to see what semi-independent modeling exercises come up with, how much they converge and diverge, etc.
https://forum.effectivealtruism.org/posts/RLWpq2oXPsunoqTEW/live-models-and-dashboards-of-anticipated-ai-wealth-going-to
https://uj-ai-wealth-philanthropy-steelman.netlify.app/global-development/
https://uj-ai-wealth-philanthropy-steelman.netlify.app/
I think it's pretty easy use AI to generate such a web page now, but I can imagine that not everyone is familiar with those tools
I see trade-offs here. There are benefits to anonymity too. I think that funders might be reluctant to reach out to those researchers if they're worried that the researchers will then treat them as a cash cow, start to bother them, and see them as having made an implied promise . Not saying that would be valid, but I can see some funders thinking that way. Researchers may be reluctant to ask difficult questions of the funders if they have a grant they are preparing or waiting on. I also suspect that some funders might be happier to take numerous anonymous questions from researchers and respond to them in bulk rather than responding to individual researchers (again, because of the fear that responding to the researcher might be interpreted as an implied commitment).
That seems helpful. I think the most critical step here would be doing something that convinces the funders that the matches and recommendations you've made are trustworthy. Once you can do that, you'll get a lot of buy-in, I would imagine.
I'd say you do not need journals. You just need credible rating and evaluation systems. Anyone can decide "I only want to read papers (better, research projects ) that got rated 70 or above." And I'm also mostly operating in environments where most researchers are already putting their research up in "working paper" or "preprint" format, and the journals are only acting as a (slow, imperfect, often unclear) signal of credibility. You can see the case at unjournal.org ... I don't mean to hijack this conversation.