All of ThomasW's Comments + Replies

People have been having similar thoughts to yours for many years, including myself. Navigating through EA epistemic currents is treacherous. To be sure, so is navigating epistemic currents in lots of other environments, including the "default" environment for most people. But EA is sometimes presented as being "neutral" in certain ways, so it feels jarring to see that it is clearly not.

Nearly everyone I know who has been around EA long enough to do things like run a university group eventually confronts the fact that their beliefs have been shaped socially... (read more)

It's also very much worth reading the linked pdf, which goes into more detail than the fact sheet.

Except, perhaps, dictators and other ne'er-do-wells.

I would guess that a significant number of power-seeking people in history and the present are power-seeking precisely because they think that those they are vying for power with are some form of "ne'er-do-wells." So the original statement:

Importantly, when longtermists say “we should try and influence the long-term future”, I think they/we really mean everyone.

with the footnote doesn't seem to mean very much. "Everyone, except those viewed as irresponsible," historically, at least, has certainly not meant everyone, and to some people means very few people.

There is for the ML safety component only. It's very different from this program in time commitment (much lower), stipend (much lower), and prerequisites (much higher, requires prior ML knowledge) though. There are a lot of online courses that just teach ML, so you could take one of those on your own and then this.

https://forum.effectivealtruism.org/posts/uB8BgEvvu5YXerFbw/intro-to-ml-safety-virtual-program-12-june-14-august-1

Sure, here they are! Also linked at the top now.

No, it is not being run again this year, sorry!

I have collected existing examples of this broad class of things on ai-improving-ai.safe.ai.

(More of a meta point somewhat responding to some other comments.)

It currently seems unlikely there will be a unified AI risk public communication strategy. AI risk is an issue that affects everyone, and many people are going to weigh in on it. That includes both people who are regulars on this forum and people who have never heard of it.

I imagine many people will not be moved by Yudkowsky's op ed, and others will be. People who think AI x-risk is an important issue but who still disagree with Yudkowsky will have their own public writing that may be partia... (read more)

There is often a clash between "alignment" and "capabilities" with some saying AI labs are pretending to do alignment while doing capabilities and others say they are so closely tied it's impossible to do good alignment research without producing capability gains.

I'm not sure this discussion will be resolved anytime soon. But I think it's often misdirected.

I think often what people are wondering is roughly "is x a good person for doing this research?" Should it count as beneficial EA-flavored research, or is it just you being an employee at a corporate AI ... (read more)

I expect that if plant based alternatives ever were to become as available, tasty, and cheap as animal products, a large proportion of people and likely nearly all EAs would become vegan. Cultural effects do matter, but in the end I expect them to be mostly downstream of technology in this particular case. Moral appeals have unfortunately had limited success on this issue.

Thanks for sharing your perspective, it's useful to hear!

I don't think the orthogonality thesis is correct in practice, and moral antirealism certainly isn't an agreed upon position among moral philosophers, but I agree that point 17 seems far fetched.

This post lines up with my outsider perspective on FHI, and it seems to be quite measured. I encourage anyone who thinks that Bostrom is really the best leader for FHI to defend that view here (anonymously, if necessary).

We should also celebrate the politicians and civil servants at the European Commission and EU Food Agency for doing the right thing. Regardless of who may have talked to them, it was ultimately up to them, and so far they've made the right choices.

A suggestion I'm throwing out just for consideration: maybe create a specific section on the frontpage for statements from organizations. I don't think there are that many organizations that want to make statements on the EA forum, but they usually seem pretty worth reading for people here. (Often: something bad happened, and here's our official stance/explanation).

A downside could be that this means organizations can be more visible than individuals about community matters. That seems possibly bad (though also how it usually works in the broader world). But it seems worse for the forum moderators to arbitrarily decide if something is important enough to be displayed somehow.

Agreed. My sense is that much of the discomfort comes from the tendancy that people have to want to have their career paths validated by a central authority. But that isn't the point of 80k. The point of 80k is to direct people towards whatever they think is most impactful. Currently that appears to be mostly x-risk.

If you meet some of the people at places like 80k and so forth, I think it's easier to realize that they are just people who have opinions and failings like anyone else. They put a lot of work into making career advising materials, and they mig... (read more)

Yes, I think it's impossible not to have norms about personal relationships (or really, anything socially important). I should perhaps have provided an example of this. Here is one:

If you move to a new place with a lot of EAs, you will likely at some point be asked if you want to live in a large group house with other EAs. These group houses are a norm, and a lot of people live  in them. This is not a norm outside of EA (though it is in maybe some other communities), so it's certainly a positive norm that has been created.

Even if EAs tended to live overwhelmingly in smaller houses, or lived with people who weren't EAs, then that would just be another norm. So I really don't think there is a way to escape norms.

I appreciate this post. But one mistake this post makes, which I think is an extremely common mistake, is assuming that there can exist a community without (soft) norms.

Every community has norms. It is impossible to get out of having norms. And so I don't think we should be averse to trying to consciously choose them.

For example, in American society it is a norm to eat meat. Sometimes this is in fact because people actively are trying to get you to eat meat. But mostly, nobody is telling other people what to eat  -- people are allowed to exercise thei... (read more)

I certainly didn't mean to imply that if you don't have one of those bullet points, you are going to be "blacklisted" or negatively affected as a result of speaking your mind. They just seemed like contributing factors for me, based on my experience. And yeah, I agree different people evaluate differently.

Thanks for sharing your perspective.

There is this, but I agree it would be good if there was one that were substantially more detailed in describing the process.

(You are probably getting downvotes because you brought up polyamory without being very specific about describing exactly how you think it relates to why Open Phil should have a public COI policy. People are sensitive about the topic, because it personally relates to them and is sometimes conflated with things it shouldn't be conflated with. Regardless, it doesn't seem relevant to your actual point, which is just that there should be a public document.)

I have not (yet) known myself to ever be negatively affected for speaking my mind in EA. However, I know others who have. Some possible reasons for the difference:

• My fundamental ethical beliefs are pretty similar to the most senior people.
• On the EA Forum, I make almost extreme effort to make tight claims and avoid overclaiming (though I don't always succeed). If I have vibes-based criticisms (I have plenty) I tend to keep them to people I trust.
• I "know my audience:" I am good at determining how to say things such that they won't be received poorly. This do

To be very clear: I am not saying "this can never be changed." I am saying that it would require changing the EA social scene-- that is, to somehow decentralize it. I am not sure how to do that well (rather than doing it poorly, or doing it in name only). But I increasingly believe it is likely to be necessary.

The problem goes beyond guardrails. Any attempts to reduce these conflicts of interest would have to contend with the extremely insular social scene in Berkeley. Since grantmakers frequently do not interact with many people outside of EA, and everyone in EA might end up applying for a grant from Open Phil, guardrails would significantly disrupt the social lives of grantmakers.

Let's not forget that you can not just improperly favor romantic partners, but also just friends. The idea of stopping Open Phil from making grants to organizations where employees ar... (read more)

I've never felt comfortable in EA broadly construed, not since I encountered it about three years ago. And yet I continue to be involved to a certain extent. Why? Because I think that doing so is useful for doing good, and many of the issues that EA focuses on are sadly still far too neglected elsewhere. Many of the people who come closest to sharing my values are in EA, so even if I didn't want to be "in EA," it would be pretty difficult to remove myself entirely.

I also love my university EA group, which is (intentionally, in part by my design, in part by... (read more)

"Living expenses while doing some of my early research" is one of the main purposes of the LTFF; to me Atlas feels like a roundabout way of getting that. LTFF asks you to have a specific high-impact project or educational opportunity for you to pursue, but as far as I know that wasn't true of Atlas.

I think The Century Fellowship would make a better comparison to the Thiel Fellowship than Atlas would. It seems aimed at similar types to the Thiel Fellowship (college age people who are prepared to start projects and need to be financially independent to do so... (read more)

In the past two years, the technical alignment organisations which have received substantial funding include

Your post does not actually say this, but when I read it I thought you were saying that these are all the organizations that have received major funding in technical alignment. I think it would have been clearer if you had said "include the following organizations based in the San Francisco Bay Area:" to make it clearer you're discussing a subset.

Anyway, here are the public numbers, for those curious, of $1 million+ grants in technical AI safety in 2... (read more)

GiveDirectly has a program for the US that you can donate to. I don't really know how good it is, but the organization in general seems excellent.

For what it's worth, I don't think this needed a retraction. It's true the original post was pretty overconfident about things. Instead of asserting something and defending it, it would probably have been better to assert it with the explicit aim of hearing criticism from people on here. That's what happened anyway, but your framing was more "here is a thing I think is good."

That would certainly be great if she would. I actually first heard about EA when I read Strangers Drowning in 2016! It's very well written.

I'm a bit confused about crossposting, are you saying it was always available? I don't remember seeing any crossposts a year ago, or being able to use the feature. In fact I used to crosspost a lot of things and specifically remember the first time I saw the crossposting feature. But maybe I just didn't notice this before.

Didn't know that about the dev teams, that's useful to know!

I still believe that there were significant problems with a section of the original statement from Max Tegmark, and they have been reinforced, not undermined, by this FAQ. To be clear, I am not referring to problems like "they left out detail x"; I am referring to the fact that a particular section was actively misleading. I understand FLI was under a lot of pressure to churn out a statement fast, so I'm not totally surprised the original statement wasn't good quality. Still, I think FLI has a responsibility not to make misleading statements that they know... (read more)

is apparently not shared by the (center-left) former Swedish government, which not only certified the Foundation as charitable but granted $30,000 in government funding and support to Nya Dagbladet in 2021

Disclaimer: I previously knew nothing about the Swedish press; I still know almost nothing. I just thought this seemed weird and spent about 20 minutes looking into it.

Some context which I think would be useful to evaluate this claim.

It appears that in Sweden the government subsidizes newspapers in the form of "press support." From reading the Wikipedia p... (read more)

You mention opportunity cost, but I think it's worth further emphasizing. To do this well, you'd need somebody who has been around a while (or at least a lot of time and cooperation from people who have). You'd need them to manage different perspectives and opinions about various things that happened. You'd need them to be a very good writer. And you'd need the writer to be someone people trust--my perspective is "Open Phil hired this person" would probably not be sufficient for trust.

There are people who could do this: Kelsey Piper is one as you suggest. ... (read more)

The latest formal iteration of which is here. You can always go through the content on your own time at the link above too.

I originally helped design the course and I ran the first iteration of a similar program. I'm not really involved with the course now but I think I'm qualified to answer. However, I did AGI safety fundamentals a long time ago and haven't done MLAB, so my knowledge of those could be wrong (though I don't think so).

In comparison to AGI Safety Fundamentals, this course a lot more technical and less conceptual. AGISF is not going to include the latest in machine learning on a technical level, and this course doesn't include as many conceptual readings.

In compa... (read more)

Thanks for writing. I will say this phenomenon isn't specific to EA. I used to organize a large ish non-EA event and huge numbers of people would fail to show up and others would try to register the day before. After enough iterations, we knew the fraction of people who did each, and we just planned for that. I wonder if you could do something similar for these events? But if it's extremely variable based on year/location, that would be harder.

Also, I realized I've been assuming that for virtual conferences, there is essentially zero downside to being a no... (read more)

Not hugely surprising, given the people at CEA have certainly thought about this more than random forum users. Still, it's good to do diligence. Thank you Eli for responding; this is a model example of somebody graciously explaining non-obvious considerations in a decision.

This is one reason I haven't wanted to live in Berkeley. Whenever I visited (which was very frequently at one point) it was pretty exhausting to have so many people running around trying to figure out and chase which things were "cool" (far from everyone in Berkeley was like this, but too many seemed to be).

Outside Berkeley I notice it much less. I still interact frequently online with a lot of people who live in Berkeley, and for me this is much more pleasant. It's a shame, because there are lots of benefits of talking to people in person. But for me they don't seem worth the costs.

[This comment isn't meant to signal any opinion about the rest of your post.]

Carlsmith's report in particular is highly interdisciplinary and draws on technical AI, economics, and also philosophy. It doesn't have much in the way of technical AI or economics claims. It's not really clear who would be most qualified to write this, but in general a philosopher doesn't seem like such a bad choice. In fact, I'd think the average philosopher with strong quantitative skills would be better at this than the average economist or certainly AI researcher.

Whether a mo... (read more)

Do I view every problem in my life and my community as analogous or bearing surprising similarity to the alignment problem

This made me laugh.

But also, as I said at the top of the post, I actually do think the alignment problem does bear surprising similarities to other things, but this is mainly because of general ideas about complex systems pertain to both.

Welcome to the forum!

I've done research in reinforcement learning, and I can say this sort of behavior is very common and expected. I was working on a project once and incorrectly programmed the reward function, leading the agent to kill itself rather than explore the environment so that it could avoid an even greater negative reward from sticking around. I didn't consider this very notable, because when I thought about the reward function, it was obvious this would happen.

Here is a spreadsheet with a really long list of examples of this kind of specificat... (read more)

The article now says the FAQ was "put together by the effective altruism forum." This is not true either, nor does it make sense. If they would like to copy and paste:

"A user on the effective altruism forum posted an FAQ."

This makes it sound a lot less official and less like something you'd want to quote in a publication. But that's because it is in fact not official at all.

I interpreted this post as the author saying that they thought general AI capabilities would be barely advanced by this kind of thing, if they were advanced by it at all. The author doesn't seem to suggest building an AGI startup, but rather some kind of AI application startup.

I'm curious if you think your reasoning applies to anything with a small chance of accelerating timelines by a small amount, or if you instead disagree with the object-level claim that such a startup would only have a small chance of accelerating timelines by a small amount.

• The Centre for Effective Altruism has had to deal with a lot of questions about Bankman-Fried since FTX’s collapse. Here’s an FAQ it put together.

This is not true. The FAQ was not put together by CEA, and so far as I know Hamish has no affiliation with CEA (nor did he ever claim to).

This is an interesting post, and I'm glad you wrote it.

People are driven by incentives, which can be created with cash in a variety of ways

I agree with these ways, but I think it's quite hard to manage incentives properly. You mention DARPA, but DARPA is a major bureacracy comprised of people who are aligned to their own incentive structures, and ultimately the most powerful organization in the world (the US government). Such a thing does not exist in AI safety -- not even close. Money would certainly help with this, but it certainly can't just be straight... (read more)

The blog post says ChatGPT is trained with proximal policy optimization. This documentation says text-davinci-003 was trained with PPO, but not text-davinci-002. 

However, it is interesting what you're saying about the request payloads, because this seems to be contradictory. So I'm not quite sure anymore. It's possible that ChatGPT was trained with PPO on top of the non-PPO text-davinci-002.

text-davinci-003 (which is effectively ChatGPT) was a bit better than text-davinci-002 anecdotally and when I benchmarked it on TriviaQA. It was only released about a week before ChatGPT so it's not necessarily unreasonable to lump them together. If you do, then the interface isn't the only change one might associate with ChatGPT.

This is an important consideration, thanks for bringing it up! I pretty much agree with all of it.