i'm confused about tithing. I yearn for the diamond emoji from GWWC, and I'm not comfortable enough to do it since I took like a 50% pay cut to do AI safety nonprofit stuff. Seems weird to make such a financial commitment, which implicates my future wife, who I have presumably not met yet, especially when I'm scraping by without too many savings per paycheck. 

Is there a sense in which I already am diamond emoji eligible, because I'm "donating 50% of my income" in the sense of opportunity cost? 50 is, famously, greater than 10. 

Some people think FTX not collapsing would've been on net worse for EA than FTX collapsing, cuz not collapsing would've led to such a grifter problem. You can find people who saw early signs of people just getting into it cuz of the free flow of money. 

I'm pretty prepared to be worried about this, if we get another couple foundations out of Anthropic alums it could be FTX all over again (without the gambling, which makes it better. But with the AI race accelerant, which makes it worse). 

Even fully autonomous weapons (those that take humans out of the loop entirely and automate selecting and engaging targets) may prove critical for our national defense. But today, frontier AI systems are simply not reliable enough to power fully autonomous weapons.

weird that one of their "red lines" is a moral line in the sand based on convictions in political philosophy, while the other one is a "not wrong but early" thing about reliability. I'm reading this as Dario pretty clearly saying that when AIs are reliable enough to have human-out-of-the-loop killchains, Anthropic will be happy to power it. 

And I'm worried this is a nuance that not all Anthropic employees or https://notdivided.org/ signers are noticing and disagree with. 

i'm being the change i want to see in the world by not listing employers/collaborators/customers/credentials in my EAG introduction channel slack post, and instead listing the things I'm responsible for. 

Lean synthesis capabilities aren't maximally elicited right now, because a lot of people view it as a text-to-text problem which leads to a bunch of stress about low amount of syntax in the pretraining data and the high code velocity (until about a year ago, language models still hadn't fully internalized migrations from Lean3 to Lean4). Techniques like Cobblestone or the stuff that Higher Order Company talks about (logic programming / language model API call hybrid architecture) seem really promising to me (and, especially as HOC points out, so much cheaper!). 

Thanks for your comment. I had broken my ankle in three places and was on too much oxycodone to engage the first time I read it. I continue to recommend your essay a lot. 

download gdoc as docx and run pandoc on it? isn't pandoc org->docx how you got it into gdocs in the first place? 

Big fan of the vid. Cried a little toward the end (when Aric says that AI2027 made him want to have the talk with his family) 

<3 I didnt know Scott was planning to donate at the time, my donation precedes Scott's--- for me, the seed was planted by Dylan and a couple folks in the EA Corner discord multiple years prior but Josh Morrison is who I asked to sit me down and push me over the edge.

Remark in guaranteed safe AI newsletter: 

Niplav writes

So, a proposal: Whenever someone claims that LLMs will d/acc us out of AI takeover by fixing our infrastructure, they will also have to specify who will pay the costs of setting up this project and running it.

I’m almost centrally the guy claiming LLMs will d/acc us out of AI takeover by fixing infrastructure, technically I’m usually hedging more than that but it’s accurate in spirit.

If transformative AI is developed soon, most open source projects (especially old ones relevant to internet infrastructure) are going to be maintained by humans with human response times. That will significantly increase the time for relevant security patches to be reviewed and merged into existing codebases, especially if at the time attackers will submit AI-generated or co-developed subtle exploits using AI systems six to nine months behind the leading capabilities, keeping maintainers especially vigilant.

I usually say we prove the patches correct! But Niplav is correct: it’s a hard social problem, many critical systems maintainers are particularly slop-phobic and won’t want synthetic code checked in. That’s why I try to emphasize that the two trust points are the spec and the checker, and the rest is relinquished to a shoggoth. That’s the vision anyway– we solve this social problem by involving the slop-phobic maintainers in writing the spec and conveying to them how trustworthy the deductive process is.

Niplav’s squiggle model: Median $~1b worth of tokens, plus all the “setting up the project, paying human supervisors and reviewers, costs for testing infrastructure & compute, finding complicated vulnerabilities that arise from the interaction of different programs…” etc costs. I think a lot’s in our action space to reduce those latter costs, but the token cost imposes a firm lower bound.

But this is an EA Forum post, meaning the project is being evaluated as an EA cause area: is it cost effective? To be cost effective, the savings from alleviating some disvalue have to be worth the money you’ll spend. As a programming best practices chauvinist, one of my pastimes is picking on CrowdStrike, so let’s not pass up the opportunity. The 2024 outage is estimated to have cost about $5b across the top 500 companies excluding microsoft. A public goods project may not have been able to avert CrowdStrike, but it’s instructive for getting a flavor of the damage, and this number suggests it could be easily worth spending around Niplav’s estimate. On cost effectiveness though, even I (who works on this “LLMs driving Hot FV Summer” thing full time) am skeptical, only because open source software is pretty hardened already. Curl/libcurl saw 23 CVEs in 2023 and 18 in 2024, which it’d be nice to prevent but really isn’t a catastrophic amount. Other projects are similar. I think a lot about the Tony Hoare quote “It has turned out that the world just does not suffer significantly from the kind of problem that our research was originally intended to solve.” Not every bug is even an exploit.

In AI safety adoption is also a reason. I expect labs to adopt stuff that they pay for, whereas research / open source public goods they'd be like "oh that's cool" then never use it.