All of Daniel_Eth's Comments + Replies

We agree there is some limit. We discuss this in the report (from footnote 26):

Physical limits come into play for a couple reasons. First, the hardware stock introduces limits in how fast improvements can be made to software. For instance, signals can only travel so fast within the hardware, and software improvements cannot occur faster than these improvements can be implemented in the hardware. Second, given a fixed stock of physical hardware, there is a (incredibly large, yet still technically) finite number of distinct algorithms that could be run on th

I think my introductory explainer on the topic is a pretty good resource for that sort of audience:
https://medium.com/@daniel_eth/ai-alignment-explained-in-5-points-95e7207300e3

OpenAI taking security more seriously seems good, and also I expect is good for reducing race dynamics (the less that US adversaries are able to hack US labs, the less tight I expect a race to be).

I think there's a decently-strong argument for there being some cultural benefits from AI-focused companies (or at least AGI-focused ones) – namely, because they are taking the idea of AGI seriously, they're more likely to understand and take seriously AGI-specific concerns like deceptive misalignment or the sharp left turn. Empirically, I claim this is true – Anthropic and OpenAI, for instance, seem to take these sorts of concerns much more seriously than do, say, Meta AI or (pre-Google DeepMind) Google Brain.

Speculating, perhaps the ideal setup would be ... (read more)

I'm pretty confident that people who prioritize their health or enjoyment of food over animal welfare can moral handshake with animal suffering vegans by tabooing poultry at the expense of beef.

Generally disagree, because the meat eaters don't get anything out of this agreement. "We'll both agree to eat beef but not poultry" doesn't benefit the meat eater. The one major possible exception imho is people in relationships – I could image a couple where one person is vegan and the other is a meat eater where they decide both doing this is a pareto-improvement.

I think it is worth at least a few hours of every person's time to help people during a war and humanitarian crisis. 

I don't think this is true, and I don't see an a priori reason to expect cause prioritization research to result in that conclusion. I also find it a little weird how often people make this sort of generalized argument for focusing on this particular conflict, when such a generalized statement should apply equally well to many more conflicts that are much more neglected and lower salience but where people rarely make this sort of argument (it feels like some sort of selective invocation of a generalizable principle).

My personal view is that being an EA implies spending some significant portion of your efforts being (or aspiring to be) particularly effective in your altruism, but it doesn't by any means demand you spend all your efforts doing so. I'd seriously worry about the movement if there was some expectation that EAs devote themselves completely to EA projects and neglect things like self-care and personal connections (even if there was an exception for self-care & connections insofar as they help one be more effective in their altruism).

It sounds like you de... (read more)

IIRC it took me about a minute or two. But I already had high context and knew how I wanted to vote, so after getting oriented I didn't have to spend time learning more or thinking through tradeoffs.

I am curious to know how many Americans were consulted about the decision to spend about $10,000 per tax-payer on upgrading nuclear weapons... surely this is a decision that American voters should have been deeply involved in, given that it impacts both their taxes and their chance of being obliterated in a nuclear apocalypse. 

I think there's a debate to be had about when it's best for political decisions be decided by what the public directly wants, vs when it's better for the public to elect representatives that make decisions based on a combination... (read more)

I don't have any strong views on whether this user should have been given a temporary ban vs a warning, but (unless the ban was for a comment which is now deleted or a private message, which are each possible, and feel free to correct me if so), from reading their public comments, I think it's inaccurate (or at least misleading) to describing them as "promoting violence". Specifically, they do not seem to have been advocating that anyone actually use violence, which is what I think the most natural interpretation of "promoting violence" would be. Instead, ... (read more)

Worth noting that in humans (and unlike in most other primates) status isn't primarily determined solely by dominance (e.g., control via coercion), but instead is also significantly influenced via prestige (e.g., voluntary deference due to admiration). While both dominance and prestige play a large role in determining status among humans, if anything prestige probably plays a larger role.

(Note – I'm not an expert in anthropology, and anyone who is can chime in, but this is my understanding given my amount of knowledge in the area.)

Note to Israelis who may be reading this: I did not upvote/downvote this post and I do not intent to vote on such posts going forward. I think you should do the same.

You're free to vote (or refrain from voting) how you want, but the suggestion to others feels illiberal to me in a way that I think is problematic. Would you also suggest that any Palestinians reading this post refrain from voting on it? (Or, going a step further, would you suggest Kenyan EAs refrain from voting on posts about GiveDirectly?) Personally, I think both Israeli EAs and Pales... (read more)

Another group that naturally could be in a coalition with those 2 – parents who just want clean air for their children to breathe from a pollution perspective, unrelated to covid. (In principle, I think may ordinary adults should also want clean air for themselves to breath due to the health benefits, but in practice I expect a much stronger reaction from parents who want to protect their children's lungs.)

My problem with the post wasn't that it used subpar prose or "could be written better", it's that it uses rhetorical techniques that make actual exchange of ideas and truth-seeking harder. This isn't about "argument style points", it's about cultivating norms in the community that make it easier for us to converge on truth, even on hard topics.

The reason I didn't personally engage with the object level is I didn't feel like I had anything particularly valuable to say on the topic. I didn't avoid saying my object-level views (if he had written a similar post with a style I didn't take issue with, I wouldn't have responded at all), and I don't want other people in the community to avoid engaging with the ideas either.

I feel like this post is doing something I really don't like, which I'd categorize as something like "instead of trying to persuade with arguments, using rhetorical tricks to define terms in such a way that the other side is stuck defending a loaded concept and has an unjustified uphill battle."

For instance:

let us be clear: hiding your beliefs, in ways that predictably leads people to believe false things, is lying. This is the case regardless of your intentions, and regardless of how it feels.

I mean, no, that's just not how the term is usually used. It's ... (read more)

That's fair. I also don't think simply putting a post on the forum is in itself enough to constitute a group being an EA group.

I don't think that's enough to consider an org an EA org. Specifically, if that was all it took for an org to be considered an EA org, I'd worry about how it could be abused by anyone who wanted to get an EA stamp of approval (which might have been what happened here – note that post is the founders' only post on the forum).

[Just commenting on the part you copied]

Feels way too overconfident. Would the cultures diverge due to communication constraints? Seems likely, though also I could imagine pathways by which it wouldn't happen significantly, such as if a singleton was already reached.

Would technological development diverge significantly, conditional on the above? Not necessarily, imho. If we don't have a self-sufficient colony on Mars before we reach "technological maturity" (e.g., with APM and ASI), then presumably no (tech would hardly progress further at all, then).

Would... (read more)

I was also surprised by how highly the EMH post was received, for a completely different reason – the fact that markets aren't expecting AGI in the next few decades seems unbelievably obvious, even before we look at interest rates. If markets were expecting AGI, AI stocks would presumably be (much more, at least compared to non-AI stocks) to the moon than they are now, and market analysts would presumably (at least occasionally) cite the possibility of AGI as the reason why. But we weren't seeing any of that, and we already knew from just general observati... (read more)

Did Eric Drexler not describe ideas like this in Engines of Creation? Either way, I would guess that Drexler has thought of similar ideas before (sans the phrase "diamondoid bacteria") and has also likely communicated these ideas to Eliezer (albeit perhaps in an informal context). Though it's also possible Eliezer came up with it independently, as it seems like a relatively natural idea to consider once you already assume diamondoid mechanosynthesis creating atomically-precise nanobots.

I don't think this is a fair representation of what happened. The only thing that Eliezer appears to have "made up" is the name "diamondoid bacteria" – the idea of diamondoid mechanosynthesis doesn't come from him, nor does the idea of applying diamondoid mechanosynthesis to building nanorobots, nor that bacteria provide a proof-of-concept for the possibility of nanorobots, nor that artificial nanorobots may share some resemblance to bacteria. Eliezer also doesn't claim to have come up with any of these ideas. You can debate the merits of any of these idea... (read more)

Granted, in principle you could also have a situation where they're less cautious than management but more cautious than policymakers and it winds up being net positive, though I think that situation is pretty unlikely. Agree the consideration you raised is worth paying attention to.

I think there are a few reasons why this is very unlikely to work in practice, at least in society today (maybe it would work if most people were longtermists of some sort):

• Focusing on such far-off issues would be seen as "silly," and would thereby hurt the political careers of those who focused on them
• Political inertia is generally strong, meaning that, by default, most things in politics don't happen, even if there's slightly more support than opposition. Here, you'd be shifting the policy proposal to one that wouldn't affect the lives of constituents, a

A union for AI workers such as data scientists, hardware and software engineers could organise labor to counterbalance the influence of shareholders or political masters.

It's not obvious to me that AI workers would want a more cautious approach than AI shareholders, AI bosses, and so on. Whether or not this would be the case seems to me to be the main crux behind whether this would be net positive or net harmful.

Do people actually think that Google+OpenAI+Anthropic (for sake of argument) are the US? Do they think the US government/military can/will appropriate those staff/artefacts/resources at some point?

I'm pretty sure what most (educated) people think is they are part of the US (in the sense that they are "US entities", among other things), that they will pay taxes in the US, will hire more people in the US than China (at least relative to if they were Chinese entities), will create other economic and technological spillover effects in greater amount in t... (read more)

In my mind there are 2 main differences:

1. Economic degrowth is undesirable, vs pausing AI is at least arguably desirable – climate change is very unlikely to lead to a literal existential catastrophe, "business as usual" tech improvements and policy changes (i.e., without overthrowing capitalism) will likely lead to a clean energy transition as is, economic degrowth would probably kill many more people than it would save, etc. Meanwhile, AI presents large existential risk in my mind, and I think a pause would probably lower this risk by a non-negligible amou

So I notice Fox ranks pretty low on that list, but if you click through to the link, they rank very high among Republicans (second to only the weather channel). Fox definitely uses rhetoric like that. After Fox (among Republicans) are Newsman and OAN, which similarly both use rhetoric like that. (And FWIW, I also wouldn't be super surprised to see somewhat similar rhetoric from WSJ or Forbes, though probably said less bluntly.)

I'd also note that the left-leaning media uses somewhat similar rhetoric for conservative issues that are supported by large groups (e.g., Trumpism in general, climate denialism, etc), so it's not just a one-directional phenomena.

I don't recall seeing many reputable publications label large-scale progressive movements (e.g., BLM, Extinction Rebellion, or #MeToo) as "uninformed mobs"

So progressive causes will generally be portrayed positively by progressive-leaning media, but conservative-leaning media, meanwhile, has definitely portrayed all those movements as ~mobs (especially for BLM and Extinction Rebellion), and predecessor movements, such as for Civli Rights, were likewise often portrayed as mobs by detractors. Now, maybe you don't personally find conservative media to b... (read more)

the piece has an underlying narrative of a covert group exercising undue influence over the government

My honest perspective is if you're an lone individual affecting policy, detractors will call you a wannabe-tyrant, if you're a small group, they'll call you a conspiracy, and if you're a large group, they'll call you an uninformed mob. Regardless, your political opponents will attempt to paint your efforts as illegitimate, and while certain lines of criticism may be more effective than others, I wouldn't expect scrutiny to simply focus on the substance eit... (read more)

either because they see the specific interventions as a risk to their work, or because they feel policy is being influenced in a major way by people who are misguided

or because they feel it as a threat to their identity or self-image (I expect these to be even larger pain points than the two you identified)

"how well resourced scientific research institutes are and thus able to get organs that they need for research."
Hmm, are they allowed to buy organs, though? Otherwise, the fact that they're well resourced might not matter much for their access to organs.

My guess is this is mostly just a product of success, and insofar as the political system increasingly takes AI X-risk seriously, we should expect to see stuff like this from time to time. If the tables were flipped and Sunak was instead pooh-poohing AI X-risk and saying things like "the safest path forward for AI is accelerating progress as fast as we can – slowing down would be Luddism" then I wouldn't be surprised to see articles saying "How Silicon Valley accelerationists are shaping Rishi Sunak’s AI plans". Doesn't mean we should ignore the negative p... (read more)

Speaking personally, I think there is a possibility of money becoming obsolete, but I also think there's a possibility of money mattering more, as (for instance) AI might allow for an easier ability to turn money into valuable labor. In my mind, it's hard to know how this all shakes out on net.

I think there are reasons for expecting the value of spending to be approximately logarithmic with total spending for many domains, and spending on research seems to fit this general pattern pretty well, so I suspect that it's prudent to generally plan to spread spen... (read more)

Annoy away – it's a good question! Of course, standard caveats to my answer apply, but there's a few caveats in particular that I want to flag:

• It's possible that by 2028 there will be one (or more) further longtermist billionaires who really open up the spigot, significantly decreasing the value of marginal longtermist money at that time
• It's possible that by 2028, AI would have gotten "weird" in ways that affect the value of money at that time, even if we haven't reached AGI (e.g., certain tech stocks might have skyrocketed by then, or it might be possible

So I'm imagining, for instance, AGIs with some shards of caring about human ~autonomy, but also other (stronger) shards that are for caring about (say) paperclips (also this was just meant as an example). I was also thinking that this might be what "a small population in a 'zoo'" would look like – the Milky Way is small compared to the reachable universe! (Though before writing out my response, I almost wrote it as "our solar system" instead of "the Milky Way," so I was imagining a relatively expansive set within this category; I'm not sure if distorted "pet" versions of humans would qualify or not.)

FWIW, I think specific changes here are unlikely to be cruxy for the decisions we make.

[Edited to add: I think if we could know with certainty that AGI was coming in 202X for a specific X, then that would be decision-relevant for certain decisions we'd face. But a shift of a few years for the 10% mark seems less decision relevant]

Another reason that the higher funding bar is likely increasing delays – borderline decisions are higher stakes, as we're deciding between higher EV grants. It seems to me like this is leading to more deliberation per grant, for instance.

Presumably this will differ a fair bit for different members of the LTFF, but speaking personally, my p(doom) is around 30%,^[1] and my median timelines are ~15 years (though with high uncertainty). I haven't thought as much about 10% timelines, but it would be some single-digit number of years.

1. ^{^}

   Though a large chunk of the remainder includes outcomes that are much "better" than today but which are also very suboptimal – e.g., due to "good-enough" alignment + ~shard theory + etc, AI turns most of the reachable universe into paperclips but leaves humans

Personally, I'd like to see more work being done to make it easier for people to get into AI alignment without becoming involved in EA or the rationality community. I think there are lots of researchers, particularly in academia, who would potentially work on alignment but who for one reason or another either get rubbed the wrong way by EA/rationality or just don't vibe with it. And I think we're missing out on a lot of these people's contributions.

To be clear, I personally think EA and rationality are great, and I hope EA/rationality continue to be on-ram... (read more)

Hmm, I think most of these grants were made when EA had much more money

I think that's true, but I also notice that I tend to vote lower on bio-related grants than do others on the fund, so I suspect there's still somewhat of a strategic difference of opinion between me and the fund average on that point.

To add to what Linch said, anecdotally, it seems like there's more disagreements when the path to impact of the grant is less direct (as opposed to, say, AI technical research), such as with certain types of governance work, outreach, or forecasting. 

In my personal opinion, the LTFF has historically funded too many bio-related grants and hasn't sufficiently triaged in favor of AI-related work.

[personal observations, could be off]

I want to add that the number tends to be higher for grants that are closer to the funding threshold or where the grant is a "bigger deal" to get right (eg larger, more potential for both upside and downside) than for those that are more obvious yes/no or where getting the decision wrong seems lower cost.

I think the idea of having AI safety conferences makes sense, but I think it would be a pretty bad idea for these conferences to be subsidized by industry. Insofar as we want to work with industry on AI safety related stuff, I think there's a lot of other stuff ahead of conferences that both: a) industry would be more excited about subsidizing, and b) I'd worry less about the COI leading to bad effects. (For instance, industry subsidies for mechanistic interpretability research.)

IDK 160% annualized sounds a bit implausible. Surely in that world someone would be acting differently (e.g. recurring donors would roll some budget forward or take out a loan)?

Presumably the first step towards someone acting differently would be the LTFF/EAIF (perhaps somewhat desperately) alerting potential donors about the situation, which is exactly what's happening now, with this post and a few others that have recently been posted.

I would be curious to hear from someone on the recipient side who would genuinely prefer $10k in hand to $14k in three mo

[Speaking in my personal capacity, not on behalf of the LTFF] I am also strongly in favor of there being an AI safety specific fund, but this is mostly unrelated to recent negative press for longtermism. My reasons for support are (primarily): a) people who aren't EAs (and might not even know about longtermism) are starting to care a lot more about AI safety, and many of them might donate to such a fund; and b) EAs (who may or may not be longtermists) may prioritize AI safety over other longtermist causes (eg biosafety), so an AI safety specific fund may fit better with their preferences.

Hmm, I was thinking not that the benefits of the first one would be higher, but that people will more likely underestimate the benefits before they go to the first one.

Thinking this through for a minute, it seems like the obvious answer would be: let people choose either (say) $500 ticket price or $1,000, and also have a note saying "If your annual income is above XYZ, we would like to ask you to choose $1,000, though this will be done on an honor-code basis. If your income is below XYZ, feel free to choose either option" (or something like that)

on the other end of the spectrum we may ask some people to pay for their ticket price in its entirety

I'm wondering how this is going to work logistically. Will CEA ask everyone to report their income, and anyone who isn't comfortable reporting it is assumed to be rich enough that they have to pay the entire cost? This outcome feels like it would be invasive. Is instead CEA going to just ask those who are publicly known to be well off to pay the entire cost? This would presumably only raise a small amount of money, and I'd worry that it would make those people who it applied to feel like they were somewhat arbitrarily being nickled-and-dimed and rub them the wrong way.

Renting out nearby restaurants seems like plausibly a good idea, though, a) that might also be quite expensive, so I'm not sure we'd actually save on costs, and b) the logistical overhead on figuring that out could be large.