Epistemic status: Speculation from two decently informed advocates armed with anecdata.
Note on process: After having some version of this conversation several times and saying, “we should probably write about this publicly,” we took the less heroic route: we recorded one of our conversations, fed the transcript into an LLM, and then substantially revised the structure, substance, and framing ourselves. We will not be sharing the transcript, as it is in...
I want to start by noting that I am amazed and somewhat in awe of what's been accomplished by the EA animal welfare movement. I am enthusiastically supportive of all campaigns that make living and dying conditions less bad for farmed animals.
I worked on such campaigns for almost 20 years, and I'm proud of that work.
That said, I do think it's a strategic error that the EA animal movement has so thoroughly moved on from advocacy for animal liberation and diet change.&n...
The swarm has made me feel AI risk in my bones for the first time
Until the Hugging Face / OpenAI swarm thing, I think I deep down felt pretty skeptical that extinction or serious loss of control scenarios were plausible in the next few years.
I’ve been lurking on LessWrong for years, and I remember being freaked out by Yudkowsky’s ‘List of Lethalities’ in 2022, and thinking that this did seem intellectually convincing, and maybe I should do something about it. But it didn...
When is it acceptable to cite an LLM?
I couldn't find any discussion or consensus about this, so I'll ask.
I am surprised to find seemingly well-thought-out articles openly citing LLMs as research. For contrast, no one (who knows how to use Wikipedia) would cite Wikipedia: you are supposed to go to its sources. That's why [citation needed] is a thing: you don't want to build on unsubstantiated BS. So I find it hard to believe that people would cite infamously hallucination/BS-prone LLMs instead, expecting some reasoning built on that to be accepted.
If the author did check for actual sources, they would surely cite them directly instead of citing the LLM, right? E.g., if Claude says X and cites source Y, then either Y is a valid source and then why care for Claude's X; or Y is not valid and then of course why care for Claude's X. In both cases, X is useless.
And if the author couldn't be bothered to check this, why should anyone care for their reasoning?
I find it particularly surprising in a community worried about AI. Are authors really swallowing whole whatever an LLM says, and thus being its conduit, without checking if it's manipulating them?
EDIT: Citaception: Turns out that LLMs also don't know how to use Wikipedia even when in "research" mode: they are happy to quote and build on any claim X in there, even when the citations in the very same Wikipedia refute X.
I think you should get the LLM to give you the citation and then cite that (ideally after checking it yourself).
I don't cite LLMs for objective facts.
In casual situations I think it's basically okay to cite an LLM if you have a good sense of what sorts of facts LLMs are unlikely to hallucinate, namely, well-established facts that are easy to find online (because they appear a lot in the training data). But for those sorts of facts, you can turn on LLM web search and it will find a reliable source for you and then you can cite that source instead.
I think it's okay to cite LLMs for things along the lines of "I asked Claude for a list of fun things to do in Toronto and here's what it came up with".
If an Anthropic data scientist in a high-profile legal case can be hoodwinked by bad citations, I don't think that it is realistic at all to think that anyone can have a "good sense of what sorts of facts LLMs are unlikely to hallucinate".
And I thought we all have heard about lists of fun things to do full of non-existent restaurants in the way to non-existent towns?