My first Fable benchmark was to one-shot turning `emacs -batch -l dunnet` into a graphical adventure game and it hit the safety guardrails bc one of the puzzles involve nitroglycerine 😭

I, er, do not think we're bottlenecked in "attention brought to the most popular politician in the US [who happens to be an advocate for AI safety]" (as opposed to, say, "attention brought to AI safety [by the most popular politician in the US]").

I very much expect a speaker at an EA conference talking about the benefits of torture would be quickly shown the door. More to the point, this goes double for a speaker saying EAs should ditch the "doing good impartially based on reason and evidence" thing and instead pick a Great Leader to never criticize and embark on a crusade to genocide the inferior races.

Is Bernie's AI sovereign wealth fund proposal good or bad for AI safety?

(I'm deliberately sidelining the details of how exactly to establish an AI SWF because I don't think it is important this early in the conversation.)

On the pro side, a SWF is a good answer to intelligence curse issues. It may also give an higher chance of AI risks ("near-term" or "long-term") being weighted into AIcos' decisions (particularly those who are supposed to be PBCs), for the same reason the LTFF and OpenAI Foundation are beneficial and dilution through IPOs might be dangerous and lead them to act more like any publicly listed company.

A more subtle pro would be that with the more immediate and intense social fear of corporate domination subsiding, this would leave room for the more complicated issues, including existential risk, to enter the public debate.

On the anti side, there has been an argument floating around that the government owning AI stocks might discourage it from taking measures that might hurt AI profits.

I think this argument is very weak, the economy is already quite dependent on AI companies doing well, and oil SWFs don't seem to have been an issue for Norway and even the Gulf taking the green transition seriously.

I'm not even sure e.g. a pause would be a bad thing economically for AI companies? Training increasingly large models is the whole reason they can't make a profit right now.

The more interesting anti argument is that, badly constructed (in particular, if the influence of the "unitary executive" is stronger than the influence of the bipartisan, deliberative congress), this could serve as a move toward greater centralization into a small group, which would neutralize most of the pros I discussed earlier. This part is certainly more worrisome considering the current levels of corruption going on.

I didn't knew about the seven principles of UU. Apparently those are:

• 1st Principle: The inherent worth and dignity of every person;
• 2nd Principle: Justice, equity and compassion in human relations;
• 3rd Principle: Acceptance of one another and encouragement to spiritual growth in our congregations;
• 4th Principle: A free and responsible search for truth and meaning;
• 5th Principle: The right of conscience and the use of the democratic process within our congregations and in society at large;
• 6th Principle: The goal of world community with peace, liberty, and justice for all;
• 7th Principle: Respect for the interdependent web of all existence of which we are a part.

Those seem quite a bit more fundamental than the ITN framework is for EA, and I'm sure there are more fundamental principles of EA that you would in fact get the same reaction if you said they should be ditched. (In fact, they would probably be basically the same ones?)

Unitarian Universalism: defers a lot more

... who are those leaders of Unitarian Universalism that UUs are deferring a lot more than EAs defer to Moskovitz, Karnofsky, MacAskill, Ord?

One area where we can see that the Lindy effect is empirically false is stock prices. If it were true, you could buy a portfolio of the 100 stocks that have gone up the most over the last 3 years, hold them for 3 years, and beat the S&P 500. But that doesn’t work. 

... your link straightforwardly show the opposite? Momentum investing is moderately profitable in the first years before reverting to the mean as the momentum subside.

Similarly, you can find plenty work on the subject on the wiki page for the Lindy effect, notably connections with Zipf's law and the Pareto distribution. (The term "Lindy effect" itself was coined by Nassim Nicholas Taleb.)

Equity research analysts and institutional investors don’t approach financial modelling or earning estimates through blind extrapolation, or by applying a rule of thumb like the Lindy effect. They think causally, often in great detail, about companies’ future performance. And, even then, accurate forecasting is really hard.

True and neither Scott nor I said otherwise. You should have a broad prior distribution and after gaining more evidence about the gears level you should update. On the other hand it is also, uh, not true that quants can ever afford to be always strictly rigorous and not using rules of thumbs of similar caliber. 

As Scott notes, the maximum-entropy heuristic if you make no hypothesis about the explanation for a statistical trend is Lindy's law, the trend continuing for as long as it continued so far. So you might expect both the Anthropic and the Jersey Mike's trend to continue until ~2028, but not until 2050.

"On the Promotion of Safe and Socially Beneficial Artificial Intelligence" by @SethBaum from 2016

How would this address the criticisms raised of METR’s methodology?

How would this not? It doesn't use the same tasks nor does it use the same human baseliner panel as the HCAST dataset.