I'm looking for a way to make sure I reliably learn about the biggest developments such as Transformers, AlphaFold, or the grokking paper. I don't currently want to spend too much time on this, so optimal frequency would probably be monthly, though weekly is fine as well.
If you have other mechanisms of staying up to date with machine learning, I'd be curious to hear about those as well.
(I accidentally asked multiple versions of this question at once.
This was because I got the following error message when submitting:
"Cannot read properties of undefined (reading 'currentUser')"
So I wrongly assumed the submission didn't work.
@moderators)
We've gotten multiple reports of this, and you're the first person to get the exact error message, thank you so much.
Curious to hear people's answers to this - it's a tough space to keep up to date in. For high-level summaries of technical developments, I like Import AI (https://jack-clark.net/) and Last Week in AI (https://lastweekin.ai/). But both can be weighted towards a policy/public affairs focus. There are likely newsletters that are better suited for purely keeping tabs on new research papers.
For an annual view of developments, I also like State of AI (https://www.stateof.ai/)
Yannic Kilcher's youtube channel profiles fairly recent papers / "ML news" events. The videos on papers are 30-60mins, so more in depth than reading an abstract, and less time consuming than reading the paper yourself. The "ML news" videos are less technical but still a good way to keep up to date on what DeepMind, Meta, NVIDIA, etc. are up to.
The Transformers paper (Attention is All You Need) was only a poster at NIPS 2017 (not even a spotlight let alone an oral presentation). I don’t know if anyone at the time predicted the impact it would have.
It’s hard to imagine a newsletter that could have picked out that paper at the time as among the most important of the hundreds included. For comparison, I think probably that at the time, there was much more hype and discussion of Hinton and students’ capsule nets (also had a NIPS 2017 paper).
I think this is generally true of ML research. It’s usually very hard to predict impact in advance. You could probably do pretty well with 6 months to a year lag though.
I will recommend the TWIML podcast which interviews a range of good researchers, but not only on the biggest stuff.
It’s hard to imagine a newsletter that could have picked out that paper at the time as among the most important of the hundreds included. For comparison, I think probably that at the time, there was much more hype and discussion of Hinton and students’ capsule nets (also had a NIPS 2017 paper).
People at the time thought it was a big deal: https://twitter.com/Miles_Brundage/status/1356083229183201281 Even the ones who were not saying it would be "radically new" or "spicy" or "this is going to be a big deal" or a "paradigm shift" were still at least asking if it might be (out of all the hundreds of things they could have been asking about but weren't).
Incidentally, I don't know if I count, but "Attention Is All You Need" was in my June 2017 newsletter & end-of-year best-of list (and capsule nets were not - I didn't like them, and still don't, it struck me as overly-hardwired and inflexible compared to existing attention methods even prior to Transformers, hardware-unfriendly, weak on toy problems, and essentially something only of interest because Hinton had been hinting at or talking about it for years; my opinion of CapsuleNets has not improved since*). So, I don't find it hard to imagine a newsletter doing it because I did it myself.
* eg as of April 2024, despite 5600+ citations, I still have found no reason to ever cite CapsuleNets on gwern.net.
Wow, that certainly is more “attention” than I remember at the time.
I think filtering on that level of hype alone would still leave you reading way too many papers.
But I can see that it might be more plausible for someone with good judgment + finger on the pulse to do a decent job predicting what will matter (although then maybe that person should be doing research themselves).
For capabilities things, https://dblalock.substack.com/ is pretty good (though some things the author is very excited about I find underwhelming).
EDIT: weekly quick summaries of papers
I subscribe to Import AI, Rohin Shah's Alignment newsletter (mostly via the LW/AF), ChinAI (weekly), Ruder's NLP (probably dead), Creative AI (annual), State of AI (annual), Larks (annual), miscellaneous blogs & subreddits (/r/machinelearning/, /r/mlscaling, /r/reinforcementlearning, /r/thisisthewayitwillbe/, being the main ones), and the 2 AKs on Twitter (Arxiv ~daily). If you need even more ML than that, well, you'd better set up an Arxiv RSS feed and drink from the firehose.
Super helpful, thanks for your answer!