63Joined Nov 2016


This is both very informative and very helpful, thank you for the advice! That does seem like a very reasonable way of thinking about the current situation, and I'm happy to see that there already exist resources that try to compile this information.

I was already referred to AISS in private, but your recommendation helped me take the step of actually applying for their coaching. Looking forward to seeing what comes of it, thanks again!

Thank you for taking the time to provide your thoughts in detail, it was extremely helpful for understanding the field better. It also helped me pinpoint some options for the next steps. For now, I'm relearning ML/DL and decided to sign up for the Introduction to ML Safety course.

I had a few follow-up questions, if you don't mind:

What are those 100 people going to do? Either start their own org, work independently or try to find some other team.

That seems like a reasonable explanation. My impression was that the field was very talent-constrained, but you make it seem like neither talent nor funding is the bottleneck. What do you think are the current constraints for AI safety work?

My confusion about the funding/private company situation stems from my (likely incorrect) assumption that AI safety solutions are not very monetizable. How would a private company focusing primarily on AI safety make a steady profit?

OpenAI, DeepMind, Anthropic -- have the safety of AI systems as an important part of their founding DNA.

I currently view OpenAI and DeepMind more like AI product companies, with "ethics and safety as considerations but not as the focus." Does this seem accurate to you? Do engineers (in general) have the option to focus mainly on safety-related projects in these companies?

Separately, I'd also like to wish the DeepMind alignment team the best of luck! I was surprised to find that it has grown much more than I thought in recent years.

Yes, and they would have been my number one picks some years ago. However, I'm no longer convinced that they are progressing AI safety measures at the same speed they're pushing for new capabilities. Intuitively it feels unsustainable (and potentially very dangerous), which is why I'm being more cautious now.

That being said, I'm very glad that both companies are putting effort into growing their safety teams, and hope that they continue to do so.

Thank you for sharing your thoughts! It's helpful to know that others have been struggling with a similar situation.

Through investigating the need and potential for projects, it seems there are vaguely two main areas for engineers:

  1. Research engineering, it seems to be essentially helping researchers build prototypes and run models as smoothly as possible.
  2. Various meta-projects that grow the talent pool or enable knowledge to be gained and shared more efficiently. The ones in the post you linked fall under this.

It seems like getting (more useful) summaries of papers and blog posts is in very high demand. I wonder if Elicit ( is useful enough to somewhat alleviate that problem already.

I also came across this list of engineering project ideas while investigating:

I'm thinking that working on one of these could be a useful entry point. It seems viable to do while studying the field itself.

Thank you for the link, I found several collections of links and more introductory information through it. This was very helpful for finding out about potentially relevant courses and opportunities.

I struggled with a similar question back when I was a student. What I've found out is that people asking this usually want to know how the applicant describes their work and approach, and how confident or passionate a person is about the things they do.

One option could be to talk about the most exciting university project/assignment that you've worked on. You could describe something that made it interesting, what you learnt from it, and explain how you handled teamwork or prioritization during it. Interesting results are a plus, but learning experiences also make for a good story.

Other options include some kind of competitive performance, or a hobby project you felt passionate about and dedicated time and energy into. Personally I would even be happy to hear about something nice you did that helped somebody else. Feel free to be open and explain what made the experience special to you.

People asking this question usually understand that new graduates' achievements don't necessarily involve work projects. So my advice would be to not worry about the context too much.

Hey Markus, thank you for sharing your thoughts!

Considering that I need to cast a relatively wide net to find even thousands of players,  the messaging about Mineralis is not very selective on purpose. I think that the overall demographic will end up being relatively random within the set of people who are gamers and enjoy JRPGs. Whether JRPG fans have a predisposition towards EA - that remains to be seen. :)

I would be super humbled if the game ends up being a success and can be considered useful for the purpose of actually teaching EA concepts. However, I would imagine that a vast majority learns better through reading articles or listening to podcasts.

Setting up a group for discussion sounds like a valuable idea. I added a personal task for that and expect to get to it late next week. I'm thinking Discord might make the most sense, given that many developers already use it anyway.

Saw this article about Patagonia's future profits being directed to climate change -related charitable activities (on the scale of $100m / year). Working with them to direct funding to CATF etc. might be extremely high-impact?

Interesting post with a good angle, thank you for taking the time to write about this!

I've also been playing around with the idea of using art or some kind of interactive medium to convey EA concepts. I completely agree that it's important that ideas are not "pushed" as EA, but rather the work allows room for the viewer to discover the benefits of these ideas for themselves.

One idea I've had that I haven't seen explored elsewhere is a video/board game about welfare and existential risk. For example, the player(s) might step into the shoes of a world leader, drafting policies to limit risks while maximizing global happiness. They would be gradually introduced to various concepts regarding existential risk, while having interesting and challenging game systems to play against.

One of the benefits with this approach is that it doesn't have to be EA branded at all, yet can introduce difficult concepts with ease and nudge people to study or discuss these topics on their own. It would also be niche enough that market validation is important, but the costs for that could greatly be reduced through building a prototype and running a crowdfunding campaign.

In fact, I had some free time last weekend and built a functional prototype with realistic data. It's obviously at a very early stage, but could be expanded in a variety of directions. In my opinion the interactivity can bring an immersive extra dimension to a work, and can help users realize new ways to think about the causality of something.

This was a great read, and just the kind of post I have been waiting for! I think that almost all of the principles should be helpful to keep in mind for anyone engaging in this kind of public discourse. In my opinion it is very important to increase the quality of communication and thus quality of knowledge across EA folks, both current and future, both internally and towards the general public; these kinds of posts would seem to help out there.

Of course, I might slightly overvalue this kind of discussion, since I don't know about the demographics of the EA community, and might be somewhat similar to the earlier self you mentioned in the post, frustrated at not being able to learn and get up to speed about the topics I care about. I don't know whether there is a demand for these kinds of information in the next generation, although it might be interesting and somewhat relevant to find out. I wonder if anyone has an idea about that?

Either way, thank you very much for taking the time to make these thoughts public, I—as well as many others, it seems—really appreciate it!

Load More