Many people live in an area (or country) where there isn't even a single AI safety organization, and can't or don't want to move. In that sense - no they can't even join an existing organization (in any level).
(I think founding an organization has other advantages over joining an existing one, but this is my top disagreement.)
I am not a cryptographer (though I do have knowledge in cryptography) and did not try to run an explicit cost-benefit analysis. Nevertheless, I think that being a researcher doing similar things to the other researchers in this field is not likely to be impactful (for various reasons, e.g., quantum resistant ciphers exist, while quantum computers [at scale] do not; and this research field is quite active so I don't think there will be any shortage of solutions in the future). I think that it is possible (but not plausible) that one could come up with an al...
Glad to hear this was helpful! In short, the answer is no, it was much less structured (though also thought of).
These decisions were made under very different circumstances, and I'm not sure there is much value in explaining how I arrived at them. Very quickly though - I chose math because I highly enjoyed it and had the time (I did consider alternatives, such as physics and CS); cyber-security was part of my military service, which did involve some choice, but it is a very different situation and would drag me into a long digression.
Apologies if this doesn't help much. Feel free to ask any other questions (publicly or privately).
Thanks Cameron. I think that I understand our differences in views. My understanding is that you argue that language agents might be a safe path (I am not sure I fully agree with this, but I am willing to be on board so far).
Our difference then is, as you say, in whether there are models which are not safe and whether this is relevant. In Section 5, on the probability of misalignment, and in your last comment, you suggest that it is highly likely that language agents are the path forward. I am not at all convinced that this is correct (e.g., I think that i...
Thanks for responding so quickly.
I think the following might be a difference in our views: I expect that people will(/are) trying to train LLM variants that are RLHFed to express agentic behavior. There's no reason to have one model to rule them all - it only makes sense to have a distinct models for short conversations and for autonomous agents. Maybe the agentic version would get a modified prompt including some background. Maybe it will be given context from memory as you specified. Do you disagree with this?
Given all of the above, I don't see a big dif...
I, for one, think that it is good that climate change was not mentioned. Not necessarily because there are no analogies and lessons to be drawn, but rather because it can more easily be misinterpreted. I think that the kind of actions and risks are much more similar to bio and nuclear, in that there are way less actors and, at least for now, it is much less integrated to day-to-day life. Moreover, in many scenarios, the risk itself is of more abrupt and binary nature (though of course not completely so), rather than a very long and gradual process. I'd be worried that comparing AI safety to climate change would be easily misinterpreted or dismissed by irrelevant claims.
Thank you, Cameron and Simon, for writing this. It articulates some thoughts I've ben pondering. However, I would like to give one pushback, which I think is fairly significant. The relevant paragraph, which summarizes what I think is a wrong assumption, is the following:
...We think this worry is less pressing than it might at first seem. The LLM in a language agent is integrated into the architecture of the agent as a whole in a way that would make it very difficult for it to secretly promote its own goals. The LLM is not prompted or otherwise informed that
Not answering the question, but I would like to quickly mention a few of the benefits of having confidence/credible intervals or otherwise quantifying uncertainty. All of these comments are fairly general, and are not specific criticisms of GiveWell's work.
Yeah, that makes sense, and is fairly clear selection bias. Since here in Israel we have a very strong tech hub and many people finishing their military service in elite tech units, I see the opposite selection bias, of people not finding too many EA (or even EA-inspired) opportunities that are of interest to them.
I failed to mention that I think your post was great, and I would also love to see (most of) these critiques flashed out.
The fact that everyone in EA finds the work we do interesting and/or fun should be treated with more suspicion.
I would like to agree with Aaron's comment and make a stronger claim - my impression is that many EAs around me in Israel, especially those coming from a strong technical background, don't find most direct EA-work very intellectually interesting or fun (ignoring its impact).
Speaking for myself, my background is mostly in pure math and in cyber-security research / software engineering. Putting aside managerial and entrepreneurial roles, it seems to...
I am extremely impressed by this, and this is a great example of the kind of ambitious projects I would love to see more of in the EA community. I have added it to the list on my post Even More Ambitious Altruistic Tech Efforts.
Best of luck!
I completely agree with everything you said (and my previous comment was trying to convey a part of this, admittedly in much less transparent way).
I simply disagree with your conclusion - it all boils down to what we have at hand. Doubling the cost-effectiveness also requires work, it doesn't happen by magic. If you are not constrained by highly effective projects which can use your resources, sure, go for it. As it seems though, we have much more resources than current small scale projects are able to absorb, and there's a lot of "left-over" resources. Thus, it makes sense to start allocating resources to some less effective stuff.
I agree with the spirit of this post (and have upvoted it) but I think it kind of obscures the really simple thing going on: the (expected) impact of a project is by definition the cost-effectiveness (also called efficiency) times the cost (or resources).
A 2-fold increase in one, while keeping the other fixed, is literally the same as having the roles reversed.
The question then is what projects we are able to execute, that is, both come up with an efficient idea, and have the resources to execute it. When resources are scarce, you really want to squeeze as...
I am not sure that there is actually a disagreement between you and Guy.
If I understand correctly, Guy says that in so far as the funder wants research to be conducted to deepen our understanding of a specific topic, the funders should not judge researchers based on their conclusions about the topic, but based on the quality and rigor of their work in the field and their contributions to the relevant research community.
This does not seem to conflict what you said, as the focus is still on work on that specific topic.
I strongly agree with this post and it's message.
I also want to respond to Jason Crawford's response. We don't necessarily need to move to a situation where everyone tries to optimize things as you suggest, but at this point it seems that almost no one tries to optimize for the right thing. I think even changing this to a few percents of entrepreneurial work or philanthropy could have tremendous effect, without losing much of the creative spark people worry we might lose, or maybe gain even more, as new directions open.
That's great, thanks!
I was aware of Anthropic, but not of the figures behind it.
Unfortunately, my impression is that most funding for such projects are around AI safety or longtermism (as I hinted in the post...). I might be wrong about this though, and I will poke around these links and names.
Relatedly, I would love see OPP/EA Funds fund (at least a seed round or equivalent) such projects, unrelated to AI safety and longtermism, or hear their arguments against that.
Thanks for clarifying Ozzie!
(Just to be clear, this post is not an attack on you or on your position, both of which I highly appreciate :). Instead, I was trying to raise a related point, which seems extremely important to me and I was thinking about recently, and making sure the discussion doesn't converge to a single point).
With regards to the funding situation, I agree that many tech projects could be funded via traditional VCs, but some might not be, especially those that are not expected to be very financially rewarding or very risky (a few examples t...
I wrote a response post Even More Ambitious Altruistic Tech Efforts, and I would love to spinoff relevant discussion there. The tl;dr is that I think we should have even more ambitious goals, and try to initiate projects that potentially have a very large direct impact (rather than focus on tools and infrastructure for other efforts).
Also, thanks for writing this post Ozzie. Despite my disagreements with your post, I mostly agree with your opinions and think that more attention should be steered towards such efforts.
I just want to add, on top of Haydn's comment to your comment, that:
You don't need the treatment and the control group to be of the same size, so you could, for instance, randomize among the top 300 candidates.
In my experience, when there isn't a clear metric for ordering, it is extremely hard to make clear judgements. Therefore, I think that in practice, it is very likely that let's say places 100-200 in their ranking seem very similar.
I think that these two factors, combined with Haydn's suggestion to take the top candidates and exclude them from the study, make it very reasonable, and of very low cost.
Last August Stijn wrote a post titled The extreme cost-effectiveness of cell-based meat R&D about this subject.
Let me quote the bottom line (emphasis mine):
This means one euro extra funding spares 100 vertebrate land animals. Including captured and aquaculture fish (also fish used for fish meal for farm animals), the number becomes an order 10 higher: 1000 vertebrate animals saved per euro.
...
Used as carbon offsetting, cell-based meat R&D has a price around 0,1 euro per ton CO2e averted.
In addition, as I wrote in a comment, I also did a back of the...
Thanks for linking this, this looks really interesting! If anyone is aware of other similar lists, or of more information about those fields and their importance (whether positive or negative), I would be interested in that.
Thanks for detailing your thoughts on these issues! I'm glad to hear that you are aware of the different problems and tensions, and made informed decisions about them, and I look forward to seeing the changed you mentioned being implemented.
I want to add one comment about to the How to plan your career article, if it's already mentioned. I think it's really great, but it might be a little bit too long for many readers' first exposure. I just realized that you have a summary on the Career planning page, which is good, but I think it might be too short. I fo...
Thanks for publishing negative results. I think that it is important to do so in general, and especially given that many other group may have relied on your previous recommendations.
If possible, I think you should edit the previous post to reflect your new findings and link to this post.
Thanks to Aaron for updating us, and thanks guzey for adding the clarification in the head of the post.
Thank you for writing this post Brian. I appreciate your choices and would be interested to hear in the future (say in a year, and even after) how things worked out, how excited will you be about your work, and if you will be able to sustain this financially.
I also appreciate the fact that you took the time to explicitly write those caveats.
I meant the difference between using the two, I don't doubt that you understand the difference between autism and (lack of) leadership. In any case, this was not main point, which is that the word autistic in the title does not help your post in any way, and spreads misinformation.
I do find the rest of the post insightful, and I don't think you are intentionally trying to start a controversy. If you really believe that this helps your post, please explain why (you haven't so far).
I don't understand how you can seriously not understand that difference between the two. Autism is a developmental disorder, which manifests itself in many ways, most of which are completely irrelevant to your post. Whereas being a "terrible leader", as you call them, is a personal trait which does not resemble autism in almost any way.
Furthermore, the word autistic in the title is not only completely speculative, but also does not help your case at all.
I think that by using that term so explicitly in your title, you spread misinformation, and with no good reason. I ask you to change the title, or let the forum moderators handle this situation.
Note from the lead moderator: We discussed a potential change to the post title, but no participants in the discussion thought that doing so was the right move.
I personally found the title confusing and annoying for some of the reasons others have mentioned, but titles don't have to help the author's case (or even make sense).
If a claim that someone had been diagnosed with a developmental disorder were being applied with no evidence to someone other than a public figure, it would clearly run afoul of our rules. But in this case, I don't think t...
Contrary to your insinuation, I never wrote that I don't understand the difference between those two. I was pointing out that Brian's argument applies to both "(autism)" and "terrible leaders".
Hey Arden, thanks for asking about that. Let me start by also thanking you for all the good work you do at 80,000 Hours, and in particular for the various pieces you wrote that I linked to at 8. General Helpful Resources.
Regarding the key ideas vs old career guide, I have several thoughts which I have written below. Because 80,000 Hours' content is so central to EA, I think that this discussion is extremely important. I would love to hear your thoughts about this Arden, and I will be glad if others could share their views as well, or even have a separate d...
Thanks for spelling out your thoughts, these are good points and questions!
With regards to potentially impactful problems in health. First, you mentioned anti-aging, and I wish to emphasize that I didn't try to assess it at any point (I am saying this because I recently wrote a post linking to a new Nature journal dedicated to anti-aging). Second, I feel that I am still too new to this domain to really have anything serious to say, and I hope to learn more myself as I progress in my PhD and work at KSM institute. That said, my impression (which is mostly b...
Thanks for your comment Michelle! If you have any other comments to make on my process (both positive and negative), I think that would be very valuable for me and for other readers as well.
Important Edit: Everything I wrote below refers only to technical cyber-security (and formal verification) roles. I don't have strong views on whether governance, advocacy or other types of work related to those fields could be impactful. My intuition is that these are indeed more promising than technical roles.
I don't see any particularly important problem that can be ...
This is a very good question and I have some thoughts about it.
Let me begin by answering about my specific situation. As I said, I have many years of experience in programming and cyber security. Given my background and connections (mostly from the army) it was fairly easy for me to find multiple companies I could work for as a contractor/part-time employee. In particular, in the past 3 years I have worked part-time in cyber security and had a lot of flexibility in my hours. Furthermore, I am certain that it is also possible to find such positions in more ...
Thanks for cross-posting this, I probably wouldn't hear about this otherwise.
I am very interested in Open Phil's model regarding the best time to donate for such causes. If anyone is aware of similar models for large donors, I would love to hear about them.
Thanks for sharing that, that sounds like an interesting plan.
A while ago I was trying to think about potential ways to have large impact via formal verification (after reading this post). I didn't give it much attention, but it looks like others and I don't see a case for this career path to be highly impactful, but I'd to love be proven wrong. I would appreciate it if you could elaborate on your perspective on this. I should probably mention that I couldn't find a reference to formal verification at agent foundations (but I didn't really read it), and Va...
With regards to FIRE, I myself still haven't figured out how this fits with my donations. In any case, I think that giving money to beggars sums up to less than $5 per month in my case (and probably even less on average), but I guess that also depends on where you live etc.
I would like to reiterate Edo's answer, and add my perspective.
First and foremost, I believe that one can follow EA perspectives (e.g. donate effectively) AND be kind and helpful to strangers, rather than OR (repeating an argument I made before in another context).
In particular, I personally don't write giving a couple of dollars in my donation sheet, and it does not affect my EA-related giving (at least not intentionally).
Additionally, they constitute such a little fraction of my other spending, that I don't notice them financially.
Despite that, I truly b...
I see, thanks for the teaser :)
I was under the impression that you have rough estimate for some charities (e.g. StrongMinds). Looking forward to see your future work on that.
Thanks for posting that. I'm really excited about HLI's work in general, and especially the work on the kinds of effects you are trying to estimate in this post!
I personally don't have a clear picture of how much $ / WELLBY is considered good (whereas GiveWell's estimates for their leading charities is around 50-100 $ / QALY). Do you have a table or something like that on your website, summarizing your results for charities you found to be highly effectively, for reference?
Thanks again!
I recently made a big career change, and I am planning to write a detailed post on this soon. In particular, it will touch this point.
I did use use Fermi calculation to estimate my impact in my career options.
In some areas it was fairly straightforward (the problem is well defined, it is possible to meaningfully estimate the percentage of problem expected to be solved, etc.). However, in other areas I am clueless as to how to really estimate this (the problem is huge and it isn't clear where I will fit in, my part in the problem is not very clear, there ar...
I think another interesting example to compare to (which also relates to Asaf Ifergan's comment) is private research institutes and labs. I think they are much more focused on specific goals, and give their researchers different incentives than academia, although the actual work might be very similar. These kinds of organizations span a long range between academia and industry.
There are of course many such example, some of which are successful and somre are probably not that much. Here are some examples that come to my mind: OpenAI, DeepMind, The Institute...
I just wanted to say that I really like your idea, and at least at the intuitive level it sounds like it could work. Looking forward to the assessment of real-world usage!
Also, the website itself looks great, and very easy to use.
Thanks for the response.
I believe this answers the first part, why GPT-3 poses an x-risk specifically.
Did you or anyone else ever write what aligning a system like GPT-3 looks like? I have to admit that it's hard for me to even have a definition of being (intent) aligned for a system GPT-3, which is not really an agent on its own. How do you define or measure something like this?
Thanks for posting this!
Here is a link to the full report: The Oxford Principles for Net Zero Aligned Carbon Offsetting
(I think it's a good practice to include a link to the original reference when possible.)
Quick question - are these positions relevant as remote positions (not in the US)?
(I wrote this comment separately, because I think it will be interesting to a different, and probably smaller, group of people than the other one.)
Thank you for posting this, Paul. I have questions about two different aspects.
In the beginning of your post you suggest that this is "the real thing" and that these systems "could pose an existential risk if scaled up".
I personally, and I believe other members of the community, would like to learn more about your reasoning.
In particular, do you think that GPT-3 specifically could pose an existential risk (for example if it falls into the wrong hands, or scaled up sufficiently)? If so, why, and what is a plausible mechanism by which it poses an x-risk?
On a...
I think that a scaled up version of GPT-3 can be directly applied to problems like "Here's a situation. Here's the desired result. What action will achieve that result?" (E.g. you can already use it to get answers like "What copy will get the user to subscribe to our newsletter?" and we can improve performance by fine-tuning on data about actual customer behavior or by combining GPT-3 with very simple search algorithms.)
I think that if GPT-3 was more powerful then many people would apply it to problems like that. I'm conc...
At some point I tried to estimate this too and got similar results. This raised several of points:
I agree that it isn't easy to quantify all of these.
Here is something you could do, which unfortunately does not take into account the changes in charities operation at different times, but is quite easy to do (all of the figures should be in real terms).
Hi Jason, thank you for giving a quick response. Both points are very reasonable.
The contest announcement post outlined "several ways an essay could substantively inform the thinking of a panelist", namely, changing the central estimate or shape of the probability distribution of AGI / AGI catastrophe, or clarifying a concept or identifying a crux.
It would be very interesting to hear if any of the submissions did change any of the panelists' (or other Open Phil employees') mind in these ways, and how so. If not, whether because you learned an unantici... (read more)