This is a special post for quick takes by Chris Leong. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Unfortunately, the EA Communications Fellowship and the EA Blog prize shut down[1]. Any new project needs to be adapted to the new funding environment.
If someone wanted to start something in this vein, what I'd suggest would be something along the lines of AI Safety Camp. People would apply with a project to be project leads and then folk could apply to these projects. Projects would likely run over a few months, part-time remote[2].
Something like this would be relatively cheap as it would be possible for someone to run this on a volunteer basis, but it might also make sense for there to be a paid organiser at a certain point.
I'd love to see the EA forum add a section titled "Get Involved" or something similar.
There is the groups directory, but it's one of only many ways that folks can get more involved, from EAGx Conferences, to Virtual Programs, 80,000 Hours content/courses to donating.
Thanks for the suggestion Chris! I'd be really excited for the Forum (or for EA.org) to have a nice page like that, and I think others at CEA agree. We did a quick experiment in the past by adding the "Take action" sidebar link that goes to the Opportunities to take action topic page, and the link got very few clicks. We try not to add clutter to the site without good reason so we removed that link for logged in users (it's still visible for logged out users since they're more likely to get value from it). Since then we've generally deprioritized it. I would like us to pick it back up at some point, though first we'd need to decide where it should live (EA.org or here) and what it should look like, design-wise.
For now, I recommend people make updates to the Opportunities to take action wiki text to help keep it up-to-date! I've done so myself a couple times but I think it would be better as a team effort. :)
Have the forum team considered running an online event to collaborate on improving wikis? I think wikis are a deeply underrated forum feature and a fantastic way for people who aren't new but aren't working in EA to directly contribute to the EA project.
I'm glad that you like the wiki! ^^ I agree that it's a nice way for people in the community to contribute.
I believe no one on the team has focused on the wiki in a while, and I think before we invest time into it we should have a more specific vision for it. But I do like the idea of collaborative wiki editing events, so thanks for the nudge! I'll have a chat with @Toby Tremlett🔹 to see what he thinks. For reference, we do have a Wiki FAQ page, which is a good starting point for people who want to contribute.
About your specific suggestion, thank you for surfacing it and including detailed context — that's quite helpful. I agree that ideally people could contribute to the wiki with lower karma. I'll check if we can lower the minimum at least. Any more substantive changes (like making a "draft" change and getting it approved by someone else) would take more technical work, so I'm not sure when we would prioritize it.
(It looks like your link to a specific quick take did work, but if you think there's a bug then let me know!)
Yeah I agree, it does feel like a thing that should exist, like there's some obvious value to it even though I got some evidence that there was low demand for it on the Forum. I think it would be faster to add to EA.org instead so perhaps we should just add a static page there.
I like that we have a list in the wiki, so that people in the EA community can help us keep the info up-to-date by editing it, but practically speaking people don't spend much time doing that.
I’m pretty bullish on having these kinds of debates. While EA is doing well at having an impact in the world, the forum has started to feel intellectually stagnant in some ways. And I guess I feel that these debates provide a way to move the community forward intellectually. That's something I've been feeling has been missing for a while.
Having a space that is intellectually-edgy, but not edge-lord maxing seems extremely valuable. Especially given how controversial some EA ideas were early on (and how controversial wild animal welfare and AI welfare still are).
In fact, I'd go further and suggest that it would be great if they were to set up their own forum. This would allow us to nudge certain discussions into an adjacent, not-explicitly EA space instead of discussing it here.
Certain topics are a poor fit for the forum because they rate high on controversy + low-but-non-zero on relevance to EA. It's frustrating having these discussions on the forum as it may turn some people off, but at the same time declaring these off-topic risks being intellectually stiffling. Sometimes things turn out to be more important than you thought when you dive into the details. So I guess I'd really love to see another non-EA space end up being the first port of call for such discussions, with the hope that only the highest quality and most relevant ideas would make it over to the EA forum.
Although I have mixed feelings on the proposal, I'm voting insightful because I appreciate that you are looking toward an actual solution that at least most "sides" might be willing to live with. That seems more insightful than what the Forum's standard response soon ends up as: rehashing fairly well-worn talking points every time an issue like this comes up.
In fact, I'd go further and suggest that it would be great if they were to set up their own forum.
Manifold already has a highly active discord, where they can discuss all the manifold-specific issues. This did not prevent the EA Forum from discussing the topic, and I doubt it would be much different if Manifold had a proper forum instead of a discord.
This is annoying because many of these discussions rate high on controversy but low on importance for EA.
It might seem low on importance for EA to you, but I suspect some people who are upset about Manifest inviting right-wing people do not consider it low-importance.
Oh, I wasn't referring to redirecting the discussions about Manifest onto a new forum. More discussions about pro-natalism or genetic engineering to improve welfare. To be clear, I was suggesting a forum associated with Manifest rather than one more narrowly associated with Manifold.
There is a world that needs to be saved. Saving the world is a team sport. All we can do is to contribute our part of the puzzle, whatever that may be and no matter how small, and trust in our companions to handle the rest. There is honor in that, no matter how things turn out in the end.
I'll post some extracts from the commitments made at the Seoul Summit. I can't promise that this will be a particularly good summary, I was originally just writing this for myself, but maybe it's helpful until someone publishes something that's more polished:
Frontier AI Safety Commitments, AI Seoul Summit 2024
The major AI companies have agreed to Frontier AI Safety Commitments. In particular, they will publish a safety framework focused on severe risks: "internal and external red-teaming of frontier AI models and systems for severe and novel threats; to work toward information sharing; to invest in cybersecurity and insider threat safeguards to protect proprietary and unreleased model weights; to incentivize third-party discovery and reporting of issues and vulnerabilities; to develop and deploy mechanisms that enable users to understand if audio or visual content is AI-generated; to publicly report model or system capabilities, limitations, and domains of appropriate and inappropriate use; to prioritize research on societal risks posed by frontier AI models and systems; and to develop and deploy frontier AI models and systems to help address the world’s greatest challenges"
"Risk assessments should consider model capabilities and the context in which they are developed and deployed" - I'd argue that the context in which it is deployed should account take into account whether it is open or closed source/weights as open-source/weights can be subsequently modified.
"They should also be accompanied by an explanation of how thresholds were decided upon, and by specific examples of situations where the models or systems would pose intolerable risk." - always great to make policy concrete"
In the extreme, organisations commit not to develop or deploy a model or system at all, if mitigations cannot be applied to keep risks below the thresholds." - Very important that when this is applied the ability to iterate on open-source/weight models is taken into account
Seoul Declaration for safe, innovative and inclusive AI by participants attending the Leaders' Session
Signed by Australia, Canada, the European Union, France, Germany, Italy, Japan, the Republic of Korea, the Republic of Singapore, the United Kingdom, and the United States of America.
"We support existing and ongoing efforts of the participants to this Declaration to create or expand AI safety institutes, research programmes and/or other relevant institutions including supervisory bodies, and we strive to promote cooperation on safety research and to share best practices by nurturing networks between these organizations" - guess we should now go full-throttle and push for the creation of national AI Safety institutes
"We recognise the importance of interoperability between AI governance frameworks" - useful for arguing we should copy things that have been implemented overseas.
"We recognize the particular responsibility of organizations developing and deploying frontier AI, and, in this regard, note the Frontier AI Safety Commitments." - Important as Frontier AI needs to be treated as different from regular AI.
Seoul Statement of Intent toward International Cooperation on AI Safety Science
Signed by the same countries.
"We commend the collective work to create or expand public and/or government-backed institutions, including AI Safety Institutes, that facilitate AI safety research, testing, and/or developing guidance to advance AI safety for commercially and publicly available AI systems" - similar to what we listed above, but more specifically focused on AI Safety Institutes which is a great.
"We acknowledge the need for a reliable, interdisciplinary, and reproducible body of evidence to inform policy efforts related to AI safety" - Really good! We don't just want AIS Institutes to run current evaluation techniques on a bunch of models, but to be actively contributing to the development of AI safety as a science.
"We articulate our shared ambition to develop an international network among key partners to accelerate the advancement of the science of AI safety" - very important for them to share research among each other
Seoul Ministerial Statement for advancing AI safety, innovation and inclusivity
Signed by: Australia, Canada, Chile, France, Germany, India, Indonesia, Israel, Italy, Japan, Kenya, Mexico, the Netherlands, Nigeria, New Zealand, the Philippines, the Republic of Korea, Rwanda, the Kingdom of Saudi Arabia, the Republic of Singapore, Spain, Switzerland, Türkiye, Ukraine, the United Arab Emirates, the United Kingdom, the United States of America, and the representative of the European Union
"It is imperative to guard against the full spectrum of AI risks, including risks posed by the deployment and use of current and frontier AI models or systems and those that may be designed, developed, deployed and used in future" - considering future risks is a very basic, but core principle
"Interpretability and explainability" - Happy to interpretability explicitly listed
"Identifying thresholds at which the risks posed by the design, development, deployment and use of frontier AI models or systems would be severe without appropriate mitigations" - important work, but could backfire if done poorly
"Criteria for assessing the risks posed by frontier AI models or systems may include consideration of capabilities, limitations and propensities, implemented safeguards, including robustness against malicious adversarial attacks and manipulation, foreseeable uses and misuses, deployment contexts, including the broader system into which an AI model may be integrated, reach, and other relevant risk factors." - sensible, we need to ensure that the risks of open-sourcing and open-weight models are considered in terms of the 'deployment context' and 'foreseeable uses and misuses'
"Assessing the risk posed by the design, development, deployment and use of frontier AI models or systems may involve defining and measuring model or system capabilities that could pose severe risks," - very pleased to see a focus beyond just deployment
"We further recognise that such severe risks could be posed by the potential model or system capability or propensity to evade human oversight, including through safeguard circumvention, manipulation and deception, or autonomous replication and adaptation conducted without explicit human approval or permission. We note the importance of gathering further empirical data with regard to the risks from frontier AI models or systems with highly advanced agentic capabilities, at the same time as we acknowledge the necessity of preventing the misuse or misalignment of such models or systems, including by working with organisations developing and deploying frontier AI to implement appropriate safeguards, such as the capacity for meaningful human oversight" - this is massive. There was a real risk that these issues were going to be ignored, but this is now seeming less likely.
"We affirm the unique role of AI safety institutes and other relevant institutions to enhance international cooperation on AI risk management and increase global understanding in the realm of AI safety and security." - "Unique role", this is even better!
"We acknowledge the need to advance the science of AI safety and gather more empirical data with regard to certain risks, at the same time as we recognise the need to translate our collective understanding into empirically grounded, proactive measures with regard to capabilities that could result in severe risks. We plan to collaborate with the private sector, civil society and academia, to identify thresholds at which the level of risk posed by the design, development, deployment and use of frontier AI models or systems would be severe absent appropriate mitigations, and to define frontier AI model or system capabilities that could pose severe risks, with the ambition of developing proposals for consideration in advance of the AI Action Summit in France" - even better than above b/c it commits to a specific action and timeline
I'm not really focused on animal rights nor do I spend much time thinking about it, so take this comment with a grain of salt.
However, if I wanted to make the future go well for animals I'd be offering free vegan meals in the Bay Area or running a conference on how to ensure that the transition to advanced AI systems goes well for animals in the Bay Area.
Reality check: Sorry for being harsh, but you're not going to end factory farming before the transition to advanced AI technologies. Max 1-2% chance of that happening. So the best thing to do is to ensure that this goes well for animals and not just humans.
I'm confused about the theory of impact for "free vegan meals in the Bay Area" idea. A few recipients might work in AI, but I don't see the link between eating a vegan meal offered for free and making more animal-friendly AI development choices.
Zachary Robinson recently stated that CEA would choose to emphasize a principles-first approach to EA. Here are my thoughts on the kinds of strategic decisions that naturally synergies with this high-level strategy:
Growth strategy: Less focused on fast growth, more focus on attracting value-aligned talent:
Eternal September effects make it hard to both grow fast and maintain high-fidelity transmission of EA principles.
Recruiting from audiences that are capable of engaging in nuanced discussions of what these principles imply
Local EA groups: More emphasis on making events attractive for long-term members to attend vs. recruiting new members:
Greater focus on advertising events in ways that bring repeat customers vs. maximising throughput
Community investment: If the aim is to build a relatively small, high-talent community instead of a mass movement, then it makes sense to shift some amount of resources from outreach to improving the effectiveness of the community.
More emphasis on epistemics improves our ability to pick the right goals and achieve those goals intelligently (rather than throwing people at the problem)
Upskilling programs such as the Introductory/Advanced Fellowship or the Precipice reading group are helpful here
It may also make sense to start some new programs or orgs focused on topics like epistemics, leadership training or conflict-resolution (I like how there's a EA mental health - I forget the name - which is running training at scale)
Mentorship programs may also help with improving the effectiveness of individuals
The community can be more effective with less people if we make progress on long-standing community issues such as the lack of low-cost EA Hub or the limited support for people trying to establish themselves in major hubs like San Fransisco or London:
It also makes more sense for the community to fix its own issues now that EA is less on the frontlines for AI Safety/x-risk (please comment if you'd like me to explain this in more detail)
Question: How can the EA community recursively self-improve?
Weirdness: A principles-first approach suggests that the community should be more tolerant of weirdness than if we were pursuing a fast-growth strategy:
It also suggests more focus on the margin with being a community rather than a professional group given that professional groups experience strong pressure to make themselves seem more respectable.
It also suggests that avoiding jargon to maximise accessibility is less of a priority
Forum debates: Running debates like this to move the community's understanding of different cause areas forward becomes more important for the principles-first approach
Additional comments:
Some of these proposals make more sense in light of the rise of cause-specific groups (many groups now focus exclusively on AI safety, Effective Animal Advocates have their own conference, Giving What We Can is doing its own movement-building for those focused on donations, particularly donations to global poverty):
If a particular cause area wants a higher rate of growth, then cause-specific groups can pursue this objective.
Similarly, cause-specific groups can choose to be more professional or more focused on developing respectability.
A lower-growth strategy makes more sense given the pummelling EA has taken in the public relations realm:
Growth would be more challenging these days
Attempting to grow really fast is more likely to spark backlash now
Recruiting top-notch folks and developing the knowledge and skills of members of the community will improve the impression that folks form about EA
A lower-growth strategy makes more sense given the reduction in available EA funding:
When there was more funding available, it made more sense to bring in lots of people so that we could rapidly proliferate projects and orgs
We also had more funding to support people who joined, so the marginal benefit from adding people was greater
There are many people in EA who either don't have the skills to directly work on high-priority areas or wouldn't enjoy having a career in these areas. Some of these people want to directly do thing rather than just Earn to Give:
A greater focus on improving the community would mean that there would be more things for these folks to do.
One underrated factor in whether to engage in community-building[1] is how likely you are to move to a hub.
I suspect that in most cases people can achieve more when they are part of a group, rather than when they are by themselves. Let's assume that your local community doesn't already provide what you need. Let's further assume that an online community isn't sufficient for your needs either:
Then you have two main options:
• If there's already a hub that provides the community that you need, then you could move there • You could try to build up the local community
There are a lot of advantages to the former. It can be quicker than trying to build up a community yourself and being in the hub will probably lead to you having more direct impact than you could have even if you managed to build up your local community quite a bit. So while either option could end up being more impactful, there's a lot of reasons why it might make sense for people who are willing to move to just focus on figuring out how to set themselves up in a hub as soon as possible.
However, there are some people who are just not going to move to a hub, because they're too rooted in their current location. My suspicion is that more of these people should be focusing on building up the community.
Since there are less opportunities outside of the hub, the opportunity cost is lower, but more importantly, someone who is planning to stay in the same location over the longer term is likely to capture more of the value from their own community-building efforts.
Obviously, this doesn't apply to everyone and there are definitely people who can have far more impact through direct work, even whilst outside of a hub, than through community building. I would just like to see more people who are planning to stay put pick up this option.
Maybe EA should try to find a compromise on the unpaid internship issue? For example, unpaid internships up to a maximum of 2 days/week being considered acceptable with the community?
This would provide additional opportunities for people to skill up, whilst ensuring that these opportunities would still be broadly accessible.
You say "find a compromise" as if this is a big and contentious issue, but I... don't really see it coming up a lot? I know Kat Woods has recently posted elsewhere about how lots of unpaid internships are being suppressed because random bystanders on the internet object to them, but I just don't actually see that happening. I would imagine that often management capacity is more of a bottleneck than pay anyway?
If we run any more anonymous surveys, we should encourage people to pause and consider whether they are contributing productively or just venting. I'd still be in favour of sharing all the responses, but I have enough faith in my fellow EAs to believe that some would take this to heart.
I think I posted in one of the threads that I have no knowledge of what private evidence Nonlinear may have, but I just realised that I actually do. I don't think it's a big enough deal for me to go back and try to track down the actual comments and edit them, but I thought it was good practise to note this on short form nonetheless.
One of the vague ideas spinning around in my head is that maybe in addition to EA which is a fairly open, loosely co-ordinated, big-tent movement with several different cause areas, there would also in value in a more selective, tightly co-ordinated, narrow movement focusing just on the long term future. Interestingly, this would be an accurate description of some EA orgs, with the key difference being that these orgs tend to rely on paid staff rather than volunteers. I don't have a solid idea of how this would work, but just thought I'd put this out there...
Oh, I would've sworn that was already the case (with the understanding that, as you say, there is less volunteering involved, because with the "inner" movement being smaller, more selective, and with tighter/more personal relationships, there is much less friction in the movement of money, either in the form of employment contracts or grants).
I suspect that it could be impactful to study say a masters of AI or computer science even if you don't really need it. University provides one of the best opportunities to meet and deeply connect with people in a particular field and I'd be surprised if you couldn't persuade at least a couple of people of the importance of AI safety without really trying. On the other hand, if you went in with the intention of networking as much as possible, I think you could have much more success.
Someone needs to be doing mass outreach about AI Safety to techies in the Bay Area.
I'm generally more of a fan of niche outreach over mass outreach, but Bay Area tech culture influences how AI is developed. If SB 1047 is defeated, I wouldn't be surprised if the lack of such outreach ended up being a decisive factor.
There's now enough prominent supporters of AI Safety and AI is hot enough that public lectures or debates could draw a big crowd. Even though a lot of people have been exposed to these ideas before, there's something about in-person events that make ideas seem real.
EA needs more communications projects.
Unfortunately, the EA Communications Fellowship and the EA Blog prize shut down[1]. Any new project needs to be adapted to the new funding environment.
If someone wanted to start something in this vein, what I'd suggest would be something along the lines of AI Safety Camp. People would apply with a project to be project leads and then folk could apply to these projects. Projects would likely run over a few months, part-time remote[2].
Something like this would be relatively cheap as it would be possible for someone to run this on a volunteer basis, but it might also make sense for there to be a paid organiser at a certain point.
Likely due to the collapse of FTX
Despite the name, AI Safety Camp is now remote.
Someone really needs to make Asterisk meetup groups a thing.
I'd love to see the EA forum add a section titled "Get Involved" or something similar.
There is the groups directory, but it's one of only many ways that folks can get more involved, from EAGx Conferences, to Virtual Programs, 80,000 Hours content/courses to donating.
A list like that could be added to the EA Handbook, which is linked on the forum sidebar
Thanks for the suggestion Chris! I'd be really excited for the Forum (or for EA.org) to have a nice page like that, and I think others at CEA agree. We did a quick experiment in the past by adding the "Take action" sidebar link that goes to the Opportunities to take action topic page, and the link got very few clicks. We try not to add clutter to the site without good reason so we removed that link for logged in users (it's still visible for logged out users since they're more likely to get value from it). Since then we've generally deprioritized it. I would like us to pick it back up at some point, though first we'd need to decide where it should live (EA.org or here) and what it should look like, design-wise.
For now, I recommend people make updates to the Opportunities to take action wiki text to help keep it up-to-date! I've done so myself a couple times but I think it would be better as a team effort. :)
Have the forum team considered running an online event to collaborate on improving wikis? I think wikis are a deeply underrated forum feature and a fantastic way for people who aren't new but aren't working in EA to directly contribute to the EA project.
I wrote a quick take a while ago about how it's probably too hard for people to edit wikis atm - I actually can't link to it but here are my quick takes: Gemma Paterson's Quick takes — EA Forum (effectivealtruism.org)
I'm glad that you like the wiki! ^^ I agree that it's a nice way for people in the community to contribute.
I believe no one on the team has focused on the wiki in a while, and I think before we invest time into it we should have a more specific vision for it. But I do like the idea of collaborative wiki editing events, so thanks for the nudge! I'll have a chat with @Toby Tremlett🔹 to see what he thinks. For reference, we do have a Wiki FAQ page, which is a good starting point for people who want to contribute.
About your specific suggestion, thank you for surfacing it and including detailed context — that's quite helpful. I agree that ideally people could contribute to the wiki with lower karma. I'll check if we can lower the minimum at least. Any more substantive changes (like making a "draft" change and getting it approved by someone else) would take more technical work, so I'm not sure when we would prioritize it.
(It looks like your link to a specific quick take did work, but if you think there's a bug then let me know!)
Ah glad the link worked. Not sure why it looked like it didn't.
Let me know if you do end up interested in doing an editing event - happy to host an in person coworking session for it in London.
Interesting. I still think it could be valuable even with relatively few clicks. You might only even need someone to click on it once.
Yeah I agree, it does feel like a thing that should exist, like there's some obvious value to it even though I got some evidence that there was low demand for it on the Forum. I think it would be faster to add to EA.org instead so perhaps we should just add a static page there.
I like that we have a list in the wiki, so that people in the EA community can help us keep the info up-to-date by editing it, but practically speaking people don't spend much time doing that.
I’m pretty bullish on having these kinds of debates. While EA is doing well at having an impact in the world, the forum has started to feel intellectually stagnant in some ways. And I guess I feel that these debates provide a way to move the community forward intellectually. That's something I've been feeling has been missing for a while.
Let Manifest be Manifest.
Having a space that is intellectually-edgy, but not edge-lord maxing seems extremely valuable. Especially given how controversial some EA ideas were early on (and how controversial wild animal welfare and AI welfare still are).
In fact, I'd go further and suggest that it would be great if they were to set up their own forum. This would allow us to nudge certain discussions into an adjacent, not-explicitly EA space instead of discussing it here.
Certain topics are a poor fit for the forum because they rate high on controversy + low-but-non-zero on relevance to EA. It's frustrating having these discussions on the forum as it may turn some people off, but at the same time declaring these off-topic risks being intellectually stiffling. Sometimes things turn out to be more important than you thought when you dive into the details. So I guess I'd really love to see another non-EA space end up being the first port of call for such discussions, with the hope that only the highest quality and most relevant ideas would make it over to the EA forum.
Although I have mixed feelings on the proposal, I'm voting insightful because I appreciate that you are looking toward an actual solution that at least most "sides" might be willing to live with. That seems more insightful than what the Forum's standard response soon ends up as: rehashing fairly well-worn talking points every time an issue like this comes up.
Considering how much skepticism there is in EA about forecasting being a high priority cause area anyway, this seems like an ok idea :)
Manifold already has a highly active discord, where they can discuss all the manifold-specific issues. This did not prevent the EA Forum from discussing the topic, and I doubt it would be much different if Manifold had a proper forum instead of a discord.
It might seem low on importance for EA to you, but I suspect some people who are upset about Manifest inviting right-wing people do not consider it low-importance.
Oh, I wasn't referring to redirecting the discussions about Manifest onto a new forum. More discussions about pro-natalism or genetic engineering to improve welfare. To be clear, I was suggesting a forum associated with Manifest rather than one more narrowly associated with Manifold.
There is a world that needs to be saved. Saving the world is a team sport. All we can do is to contribute our part of the puzzle, whatever that may be and no matter how small, and trust in our companions to handle the rest. There is honor in that, no matter how things turn out in the end.
hear hear 👏🏼👏🏼
I'll post some extracts from the commitments made at the Seoul Summit. I can't promise that this will be a particularly good summary, I was originally just writing this for myself, but maybe it's helpful until someone publishes something that's more polished:
Frontier AI Safety Commitments, AI Seoul Summit 2024
The major AI companies have agreed to Frontier AI Safety Commitments. In particular, they will publish a safety framework focused on severe risks: "internal and external red-teaming of frontier AI models and systems for severe and novel threats; to work toward information sharing; to invest in cybersecurity and insider threat safeguards to protect proprietary and unreleased model weights; to incentivize third-party discovery and reporting of issues and vulnerabilities; to develop and deploy mechanisms that enable users to understand if audio or visual content is AI-generated; to publicly report model or system capabilities, limitations, and domains of appropriate and inappropriate use; to prioritize research on societal risks posed by frontier AI models and systems; and to develop and deploy frontier AI models and systems to help address the world’s greatest challenges"
"Risk assessments should consider model capabilities and the context in which they are developed and deployed" - I'd argue that the context in which it is deployed should account take into account whether it is open or closed source/weights as open-source/weights can be subsequently modified.
"They should also be accompanied by an explanation of how thresholds were decided upon, and by specific examples of situations where the models or systems would pose intolerable risk." - always great to make policy concrete"
In the extreme, organisations commit not to develop or deploy a model or system at all, if mitigations cannot be applied to keep risks below the thresholds." - Very important that when this is applied the ability to iterate on open-source/weight models is taken into account
https://www.gov.uk/government/publications/frontier-ai-safety-commitments-ai-seoul-summit-2024/frontier-ai-safety-commitments-ai-seoul-summit-2024
Seoul Declaration for safe, innovative and inclusive AI by participants attending the Leaders' Session
Signed by Australia, Canada, the European Union, France, Germany, Italy, Japan, the Republic of Korea, the Republic of Singapore, the United Kingdom, and the United States of America.
"We support existing and ongoing efforts of the participants to this Declaration to create or expand AI safety institutes, research programmes and/or other relevant institutions including supervisory bodies, and we strive to promote cooperation on safety research and to share best practices by nurturing networks between these organizations" - guess we should now go full-throttle and push for the creation of national AI Safety institutes
"We recognise the importance of interoperability between AI governance frameworks" - useful for arguing we should copy things that have been implemented overseas.
"We recognize the particular responsibility of organizations developing and deploying frontier AI, and, in this regard, note the Frontier AI Safety Commitments." - Important as Frontier AI needs to be treated as different from regular AI.
https://www.gov.uk/government/publications/seoul-declaration-for-safe-innovative-and-inclusive-ai-ai-seoul-summit-2024/seoul-declaration-for-safe-innovative-and-inclusive-ai-by-participants-attending-the-leaders-session-ai-seoul-summit-21-may-2024
Seoul Statement of Intent toward International Cooperation on AI Safety Science
Signed by the same countries.
"We commend the collective work to create or expand public and/or government-backed institutions, including AI Safety Institutes, that facilitate AI safety research, testing, and/or developing guidance to advance AI safety for commercially and publicly available AI systems" - similar to what we listed above, but more specifically focused on AI Safety Institutes which is a great.
"We acknowledge the need for a reliable, interdisciplinary, and reproducible body of evidence to inform policy efforts related to AI safety" - Really good! We don't just want AIS Institutes to run current evaluation techniques on a bunch of models, but to be actively contributing to the development of AI safety as a science.
"We articulate our shared ambition to develop an international network among key partners to accelerate the advancement of the science of AI safety" - very important for them to share research among each other
https://www.gov.uk/government/publications/seoul-declaration-for-safe-innovative-and-inclusive-ai-ai-seoul-summit-2024/seoul-statement-of-intent-toward-international-cooperation-on-ai-safety-science-ai-seoul-summit-2024-annex
Seoul Ministerial Statement for advancing AI safety, innovation and inclusivity
Signed by: Australia, Canada, Chile, France, Germany, India, Indonesia, Israel, Italy, Japan, Kenya, Mexico, the Netherlands, Nigeria, New Zealand, the Philippines, the Republic of Korea, Rwanda, the Kingdom of Saudi Arabia, the Republic of Singapore, Spain, Switzerland, Türkiye, Ukraine, the United Arab Emirates, the United Kingdom, the United States of America, and the representative of the European Union
"It is imperative to guard against the full spectrum of AI risks, including risks posed by the deployment and use of current and frontier AI models or systems and those that may be designed, developed, deployed and used in future" - considering future risks is a very basic, but core principle
"Interpretability and explainability" - Happy to interpretability explicitly listed
"Identifying thresholds at which the risks posed by the design, development, deployment and use of frontier AI models or systems would be severe without appropriate mitigations" - important work, but could backfire if done poorly
"Criteria for assessing the risks posed by frontier AI models or systems may include consideration of capabilities, limitations and propensities, implemented safeguards, including robustness against malicious adversarial attacks and manipulation, foreseeable uses and misuses, deployment contexts, including the broader system into which an AI model may be integrated, reach, and other relevant risk factors." - sensible, we need to ensure that the risks of open-sourcing and open-weight models are considered in terms of the 'deployment context' and 'foreseeable uses and misuses'
"Assessing the risk posed by the design, development, deployment and use of frontier AI models or systems may involve defining and measuring model or system capabilities that could pose severe risks," - very pleased to see a focus beyond just deployment
"We further recognise that such severe risks could be posed by the potential model or system capability or propensity to evade human oversight, including through safeguard circumvention, manipulation and deception, or autonomous replication and adaptation conducted without explicit human approval or permission. We note the importance of gathering further empirical data with regard to the risks from frontier AI models or systems with highly advanced agentic capabilities, at the same time as we acknowledge the necessity of preventing the misuse or misalignment of such models or systems, including by working with organisations developing and deploying frontier AI to implement appropriate safeguards, such as the capacity for meaningful human oversight" - this is massive. There was a real risk that these issues were going to be ignored, but this is now seeming less likely.
"We affirm the unique role of AI safety institutes and other relevant institutions to enhance international cooperation on AI risk management and increase global understanding in the realm of AI safety and security." - "Unique role", this is even better!
"We acknowledge the need to advance the science of AI safety and gather more empirical data with regard to certain risks, at the same time as we recognise the need to translate our collective understanding into empirically grounded, proactive measures with regard to capabilities that could result in severe risks. We plan to collaborate with the private sector, civil society and academia, to identify thresholds at which the level of risk posed by the design, development, deployment and use of frontier AI models or systems would be severe absent appropriate mitigations, and to define frontier AI model or system capabilities that could pose severe risks, with the ambition of developing proposals for consideration in advance of the AI Action Summit in France" - even better than above b/c it commits to a specific action and timeline
https://www.gov.uk/government/publications/seoul-ministerial-statement-for-advancing-ai-safety-innovation-and-inclusivity-ai-seoul-summit-2024
I'm not really focused on animal rights nor do I spend much time thinking about it, so take this comment with a grain of salt.
However, if I wanted to make the future go well for animals I'd be offering free vegan meals in the Bay Area or running a conference on how to ensure that the transition to advanced AI systems goes well for animals in the Bay Area.
Reality check: Sorry for being harsh, but you're not going to end factory farming before the transition to advanced AI technologies. Max 1-2% chance of that happening. So the best thing to do is to ensure that this goes well for animals and not just humans.
Anyway, that concludes my hot-take.
There is an AI, Animals, & Digital Minds conference that's being planned in the Bay Area for earlyish 2025! Updates will be announced in the AI & Animals newsletter.
I'm confused about the theory of impact for "free vegan meals in the Bay Area" idea. A few recipients might work in AI, but I don't see the link between eating a vegan meal offered for free and making more animal-friendly AI development choices.
Presumably you’d be doing outreach at the same time to influence values.
What could principles-first EA look like?
Zachary Robinson recently stated that CEA would choose to emphasize a principles-first approach to EA. Here are my thoughts on the kinds of strategic decisions that naturally synergies with this high-level strategy:
Additional comments:
To Community Build or Not
One underrated factor in whether to engage in community-building[1] is how likely you are to move to a hub.
I suspect that in most cases people can achieve more when they are part of a group, rather than when they are by themselves. Let's assume that your local community doesn't already provide what you need. Let's further assume that an online community isn't sufficient for your needs either:
Then you have two main options:
• If there's already a hub that provides the community that you need, then you could move there
• You could try to build up the local community
There are a lot of advantages to the former. It can be quicker than trying to build up a community yourself and being in the hub will probably lead to you having more direct impact than you could have even if you managed to build up your local community quite a bit. So while either option could end up being more impactful, there's a lot of reasons why it might make sense for people who are willing to move to just focus on figuring out how to set themselves up in a hub as soon as possible.
However, there are some people who are just not going to move to a hub, because they're too rooted in their current location. My suspicion is that more of these people should be focusing on building up the community.
Since there are less opportunities outside of the hub, the opportunity cost is lower, but more importantly, someone who is planning to stay in the same location over the longer term is likely to capture more of the value from their own community-building efforts.
Obviously, this doesn't apply to everyone and there are definitely people who can have far more impact through direct work, even whilst outside of a hub, than through community building. I would just like to see more people who are planning to stay put pick up this option.
Here I'm using community-building in a broad sense.
Is anyone doing broad AI Safety outreach to techies in the Bay Area?
It seems very important to have a group doing this given how much opinions within Bay Area tech influence how AI is developed.
If SB 1047 doesn't pass, this ball being dropped may be partially to blame.
Maybe EA should try to find a compromise on the unpaid internship issue? For example, unpaid internships up to a maximum of 2 days/week being considered acceptable with the community?
This would provide additional opportunities for people to skill up, whilst ensuring that these opportunities would still be broadly accessible.
(In countries where this is legally allowed)
You say "find a compromise" as if this is a big and contentious issue, but I... don't really see it coming up a lot? I know Kat Woods has recently posted elsewhere about how lots of unpaid internships are being suppressed because random bystanders on the internet object to them, but I just don't actually see that happening. I would imagine that often management capacity is more of a bottleneck than pay anyway?
If we run any more anonymous surveys, we should encourage people to pause and consider whether they are contributing productively or just venting. I'd still be in favour of sharing all the responses, but I have enough faith in my fellow EAs to believe that some would take this to heart.
I think I posted in one of the threads that I have no knowledge of what private evidence Nonlinear may have, but I just realised that I actually do. I don't think it's a big enough deal for me to go back and try to track down the actual comments and edit them, but I thought it was good practise to note this on short form nonetheless.
One of the vague ideas spinning around in my head is that maybe in addition to EA which is a fairly open, loosely co-ordinated, big-tent movement with several different cause areas, there would also in value in a more selective, tightly co-ordinated, narrow movement focusing just on the long term future. Interestingly, this would be an accurate description of some EA orgs, with the key difference being that these orgs tend to rely on paid staff rather than volunteers. I don't have a solid idea of how this would work, but just thought I'd put this out there...
Oh, I would've sworn that was already the case (with the understanding that, as you say, there is less volunteering involved, because with the "inner" movement being smaller, more selective, and with tighter/more personal relationships, there is much less friction in the movement of money, either in the form of employment contracts or grants).
I suspect that it could be impactful to study say a masters of AI or computer science even if you don't really need it. University provides one of the best opportunities to meet and deeply connect with people in a particular field and I'd be surprised if you couldn't persuade at least a couple of people of the importance of AI safety without really trying. On the other hand, if you went in with the intention of networking as much as possible, I think you could have much more success.
Someone needs to be doing mass outreach about AI Safety to techies in the Bay Area.
I'm generally more of a fan of niche outreach over mass outreach, but Bay Area tech culture influences how AI is developed. If SB 1047 is defeated, I wouldn't be surprised if the lack of such outreach ended up being a decisive factor.
There's now enough prominent supporters of AI Safety and AI is hot enough that public lectures or debates could draw a big crowd. Even though a lot of people have been exposed to these ideas before, there's something about in-person events that make ideas seem real.