Hide table of contents

Post status: First-pass, looking for feedback, aiming to build and share something more polished and comprehensive


The Unjournal: background, progress, push for communications and impact

The Unjournal is a nonprofit that publicly evaluates and rates research, focusing on impact.

We now have about 30 "evaluation packages" posted (here,  indexing in scholarly ecosystem). Each package links the (open-access) papers, and contains 1-3 expert evaluations and ratings of this work, as well as a synthesis and the evaluation manager's report. Some also have author responses. We're working to make this content more visible and more useful, including through accessible public summaries.  
 

Also see: 

 

Notebook/chatbot exploration (esp. NotebookLM)

We're considering/building another tool: a notebook/chatbot that will enable you to ask questions about the research and the evaluations. We're trialing a few approaches (such as engineering with anythingLLM etc.), and wanted to get early thoughts and opinions. 

NotebookLM seems particularly easy to set up, and seems to be yielding some positive results. This tool got some eyeballs in academic social media for its AI-generated podcasts of research (aimed at a lay audience, cheerleading, some inaccuracies; e.g., see here and here, with caution), but its notebook chat feature seems more useful for us. 

We can upload or scrape a range of sources, including the research paper itself, the evaluations, responses, and syntheses, and even related podcast transcripts and blogs.

 

For "Lead Exposure..." (Crawfurd et al, CGDEV); shared paper, evaluations, blog 

 

It seems to give fairly useful and accurate answers. It sometimes mixes up things like ‘what we suggest for future evaluators to do’ and ‘what the evaluators actually wrote’. But it tracks and links the sources for each answer, so you can double-check it pretty easily. 

Each reference note links and highlights the source content

 

It gets technical and seems clever. Hovering over a footnote reveals the supporting info

 

It suggests questions to ask, and can generate a chatty podcast

 

Some limitations: The formatting and UX also leave a bit to be desired (e.g., you need to click 'notebook guide' a lot). It can be hard to see exactly what the referenced content is going; especially if it comes from a scraped website, the formatting can be weird. I don't see an easy way to upload or download content in bulk. Saved 'notes' lose the links to the references. 

 

So far we've made notebooks


To request access

At the moment you can't share the notebooks publicly –  if you share a non-institutional email we can give you access to specific notebooks, as we create them. To request access Complete the 3-question request form here.
 

Would like your feedback (thanks!)

We've only created a few of these, we could create more without too much effort. But before we dive in, looking for suggestions on:

  • Are these useful? How could they be made more useful?
  • Is NotebookLM the best tool here? What else should we consider?
  • How to best automate this process?
  • Any risks we might not be anticipating?

8

0
0

Reactions

0
0

More posts like this

Comments
No comments on this post yet.
Be the first to respond.
Curated and popular this week
Relevant opportunities