There is a limit to how much a for-profit company can care about safety. It is of existential importance to make profit, so you have to sell your product. It's just how it is.

OpenAI and Anthropic are technically public benefit companies that have other objectives than just profit, but to my understanding they are formulated in a way that makes them not relevant currently, so profit is more important.

I don't say that there aren't people in these companies that care about safety, or that safety is completely ignored when developing products, just that there is a limit to how far the companies will consider safety.

That's marketing.

Anthropic and OpenAI have been quite successful at promoting the view that their models are very capable or even dangerous. By exaggerating the risks of these models they create a reputation as morally conscious, safety-oriented companies that nevertheless have the best models that everyone should be aware of. This gets the sympathies of investors and AI safety people alike.

But of course they must sell their models to get their money back.

I often hear the suggestion that people should short stock when they don't believe in a company. I don't think that it is a very good piece of advice.

Shorting is notoriously difficult and carries the possibility of unlimited loss. Even if you believe that a stock will crash, small errors such as timing the crash one year too early or using a miscalculated stop order to stop shorting too early can lead to massive losses. Determining the actual risk involved is not very straightforward. Shorting is often accompanied by risk-managing tactics such as hedging that also require careful considerations.

Due to all this, even if you believe a stock is going to crash, shorting is usually a bad idea. Basically, the skillset required to short properly is different from the skills used to predict whether or not AI will succeed commercially or not. An expert in AI can do the latter prediction, but the former requires you to also be specialized in complex investments. If you are "only" an AI expert, bets offer a much more managed risk profile.

In the context of my post, the term "narrative argument" was really clearly used to refer to fictional stories embedded in a text as part of its argument, such as AI2027's short story or the parables in Y&S's book. My text simply does not have that.

By completely misunderstanding even basic concepts such as "narrative argument", your comment was just a snarky jab and had little value. It didn't comment the actual content of my text, instead, and largely unjustified by any real analysis, it tried to "get back at me" by blaming me for things I blame others for, like non-evidence based argumentation and narrative arguments.

Since I find it difficult to believe that you really don't understand what a narrative argument is, I reasonably count a high probability for you simply acting in bad faith.

I don't know if "snitching" you is allowed or not, and I apologize for the moderators if it's not, but I think using examples of bad behavior is relevant in discussions of what is allowed on the forum.

I feel that there could be greater transparency in how these moderation decisions are done. In my opinion, although Yarrow has been quite confrontational at times, they haven't really done anything that would necessitate such a punishment. While the forum is allowed to moderate its users in any way it likes, one shouldn't be rate-limited just because they have disagreements, especially if their opinions go against the mainstream opinion on the forum. This puts the neutrality of the forum moderation at question.

Personally, I have pretty bad experiences writing on this forum. I get highly-upvoted comments from people who seemingly haven't even read what I wrote and instead throw in bad-faith arguments, which I find borderline insulting and certainly unwelcoming. See, for example, this comment that was given as a reply to my text, where I pointed out that many sources use narrative arguments (stories, parables, etc.) instead of technical evidence. The commenter accused me of writing a narrative argument... which is simply not true, as my text was not written in a form of a story. Either they didn't read at all what I wrote or intentionally misunderstood it. And that is the most upvoted comment!

In general, the quality of discussion on this forum is very low. I feel it is pretty harsh to say this, but it is my experience and it strongly discourages me from participating at all. In this atmosphere, rate-limiting Yarrow appears not as an impartial decision, since many other, much more problematic comments appear in the forum all the time.

I don't really see why anyone would use AI other than maybe translate phrases if one does not know English (or the language they are writing) well.

Language models add unnecessary fluff to the text that takes space away from actual content. I'd ask the following question: is the text written by the AI longer than the prompt? If the answer is yes, then please do not use that AI text. All extra text written by the model is fluff and meaningless.

Language models are also really bad at writing. They tend to overuse stylistic devices to the point that the text is heavy and difficult to read. They split the text into unnatural chapters, tend to add more subheadings than necessary, overuse tables and lists, and in general write horribly. Some people claim that AI helps people who are bad at writing to be able to take part in discussions, and that argument really makes no sense since AI cannot write either and I'd rather read bad English than fluff. The only exception to this is using AI for translation, since that is obviously a real blocker.

Lastly, there is the question of effort. To put an effort into writing is a proof that someone cares about this enough to write it down, so I should also care about reading it. If I recognize that a text is AI written, my motivation to go through it drops immediately. In a world of constant spam and sensory – and literary – overload, my time is valuable, and effort is the currency that gets me to read your text.

Your argument is very similar to creationist and other pseudoscientific/conspiracy theory-style arguments.

A creationist might argue that the existence of life, humanity, and other complex phenomena is "evidence" for intelligent design. If we allow this to count as "limited" evidence (or whatever term we choose to use), it is possible to follow through a Pascal's wager-style argument and posit that this "evidence", even if it has high uncertainty, is enough to merit an action.

It is always possible to come up with "evidence" for any claim. In evidence-based decision making, we must set a bar for evidence. Otherwise, the word "evidence" would lose it's meaning, and we'd be wasting our resources considering every piece of knowledge there exists as "evidence".

You could have moderate-quality studies that withstand scutiny

If the studies withstand scrutiny, then they are high-quality studies. Of course, it is possible that the study has multiple conclusions, and some of them are undermined by scrutiny and some are not, or that there are errors that do not undermine the conclusions. These studies can of course be used as evidence. I used "high-quality" as the opposite of "low-quality", and splitting hair about "moderate-quality" is uninteresting.

you could have preliminary studies which are suggestive but which haven't been around long enough for scrutiny to percolate up

This is a good basis when, e.g., funding new research, as confirming and replicating recent studies is an important part of science. In this case, it doesn't matter that much if the study's conclusions end up being true or false, as confirming either way is valuable. Researching interesting things is good, and even bad studies are evidence that the topic is interesting. But they are not evidence that should be used for other kind of decision-making.

The fact that you are using the word "evidence" in this way is causing you to misinterpret the quoted statement.

You are again splitting a hair about the meanings of words. The important thing is that they are advocating for making decisions without sufficient evidence, which is something I oppose. Their report is long and contains many AI risks, some of which (like deepfakes) have high-quality studies behind them, while others (like X-risks) do not. As a whole, the report "has some evidence" that there are risks associated with AI. So they talk about "limited evidence". What is important is that they imply this "limited evidence" is not sufficient for making decisions.

But "limited evidence" isn't the same as near-zero evidence

Splitting a hair. You can call your evidence limited evidence if you want. It won't get you a free pass that your argument should be considered. If it has too much uncertainty or doesn't withstand scrutiny, it shouldn't be taken in as evidence. Otherwise we end up in the creationist situation.

If you believe that evidence that does not withstand scrutiny (that is, evidence that does not meet basic quality standards, contains major methodological errors, is statistically insignificant, is based on fallacious reasoning, or any other reason why the evidence is scrutinized) is evidence that we should use, then you are advocating for pseudoscience. The expected value of benefits based on such evidence is near zero.

I'm sorry if criticizing pseudoscience is frustrating, but that kind of thinking has no place in rational decision-making.

Your summary of the quoted text is inaccurate. You claim that this is an arguement that evidence is not something that in inherently required, but the quote says no such thing. Instead, it references "a large body of scientific evidence" and "stronger evidence" vs "limited evidence". This quote essential makes the same arguement I do above. How can we square the differences in these interpretations?

The quoted text implies that the evidence would not be sufficient under normal circumstances, hence the "evidence dilemma". If the amount of evidence was sufficient, there would be no question about what is the correct action. While the text washes its hands from making the actual decision to rely on insufficient evidence, it clearly considers this as a serious possibility, which is not something that I believe anyone should advocate.

You are splitting hairs about the difference between "no evidence" and "limited evidence". The report considers a multitude of different AI risks, some of which have more evidence and some of which have less. What is important is that they bring up the idea that policy should be made without proper evidence.

The "scientific" phrasing frustrates me because I feel like it is often used to suggest high rigor without actually demonstrating that such rigor actually applies to a give situation, and because I feel like it is used to exclude certain categories of evidence when those categories are relevant, even if they are less strong compared to other kinds of evidence. I think we should weigh all relevant evidence, not exclude cetain pieces because they aren't scientific enough.

Again, you are attacking me because of the word "scientific" instead of attacking my arguments. As I have many, many times said, studies should be weighted based on their content and the scrutiny it receives. To oppose the word "science" just because of the word itself is silly. Your idea that works are arbitrarily sorted to "scientific" and "non-scientific" based on "style points" instead of assessing their merits is just wrong and a straw-man argument.

I don't think my argument leads to this conclusion. I'm just saying that AI risk has some evidence behind it, even if it isn't the most rigorous evidence! That's why I'm being such a stickler about this! If it were true that AI risk has actually zero evidence then of course I wouldn't buy it! But I don't think there actually is zero evidence even if AI risk advocates sometimes overestimate the strength of the evidence.

Where have I ever claimed that there is no evidence worth considering? In the start of my post, I write:

What unites many of these statements is the thorough lack of any evidence.

There are some studies that are rigorously conducted that provide some meager evidence. Not really enough to justify any EA intervention. But instead of referring to these studies, people use stuff like narrative arguments and ad-hoc models, which have approximately zero evidential value. That is the point of my post.

What about these statements makes you think that I don't believe uncertainty affects decision making? It seems like I say that it does affect decision making in my comment.

If you believe this, I don't understand where you disagree with me, other than you weird opposition to the word "scientific".

I think many people in the EA community in fact have this view. Do you think those people should still prefer GHD because AI is off limits due to not being "scientific"? I would consider this to be "for style points", and disagree with this approach.

It seems you have an issue with the word "scientific" and are constructing a straw-man argument around it. This has nothing to do with "style points". As I have already explained, by scientific I only refer to high-quality studies that withstand scrutiny. If a study doesn't, then it's value as evidence is heavily discounted, as the probability of the conclusions of the study being right despite methodological errors, failures to replicate it, etc. is lower than if the study does not have these issues. If a study hasn't been scrutinized at all, it is likely bad, because the amount of bad research is greater than the amount of good research (for example, if we look at the rejection rates of journals/conferences), and lack of scrutiny implies lack of credibility as researchers do not take the study seriously enough to scrutinize it.

The conclusion that cause A is preferable to cause B involves the uncertainty about both causes. Even if cause A has more rigorous evidence than cause B, that doesn't mean the conclusion that benefits(A) > benefits(B) is similarly rigorous.

Yet E[benefits(A)] > E[benefits(B)] is a rigorous conclusion, because the uncertainty can be factored into the expected value.

Can I ask why? Do you think AI won't be a "big deal" in the reasonably near future?

The International AI Safety Report lists many realistic threats (the first one of those is deepfakes, to give an example). Studying and regulating these things is nice, but they are not effective interventions in terms of lives saved etc.

I'm really at a loss here. If your argument is taken literally, I can convince you to fund anything, since I can give you highly uncertain arguments for almost everything. I cannot believe this is really your stance. You must agree with me that uncertainty affects decision making. It only seems that the word "scientific" bothers you for some reason, which I cannot really understand either. Do you believe that methodological errors are not important? That statistical significance is not required? That replicability does matter? To object to the idea that these issues cause uncertainty is absurd.