I recently completed the AGI Safety Fundamentals course (on AI alignment) and really enjoyed it.

I imagine lots of people on the forum have taken the course and now would be a good time to share reflections. I’m hoping this thread might be particularly useful for people considering taking it in future. 

I’d be interested to hear many different kinds of reflections, but here are some prompts:

  1. What do you think you’ve learnt? How has it impacted your plans, if it has?
  2. Did you think it was worth the time?
  3. Who would you recommend it to?
  4. Do you wish you had done something differently, or known something in advance? (Any other advice?)

I’ll leave my own reflections in the comments.

27

0
0

Reactions

0
0
Comments7


Sorted by Click to highlight new comments since:
  • As a result of AGI SF readings and other sporadic AI safety readings… 
    • … I feel more confident asking questions of people who know more than I do
      • I feel like I know the vocabulary, main threat scenarios, and rough approaches to solving the problem, such that I can situate new facts into existing taxonomies 
    • … I’m better able to tell when prominent people disagree / things have more texture
  • Some (self-)critiques 
    • Honestly thought content for some of the weeks was a bit weak if you just wanted an overview of the alignment problem (e.g., adversarial techniques for scalable oversight probably isn’t what you need to understand if you’re trying to assess the risk) 
    • I wish I’d set up strong accountability
    • Wish it had more counter-arguments to classic AI risk. 
      • Apparently Stanford AI group modified curriculum to have more of these? 
    • Wish I’d been more active in discussion or questions beforehand: give yourself a chance to be wrong!
  • Tips I’d recommend for learning more about AI risk
    • Just start! Always feels daunting to dive into but just find a few explainer articles and dive in
    • Spaced repetition for learning really does go hard
    • Talk to knowledgeable people who give you space to be wrong
    • Write up your thoughts and have knowledgeable people poke holes/ show you where you’re missing something 
  • A few resources that I keep coming back to (not from AGI SF): 

Thank you for making this thread Clifford, and we're really grateful for all feedback! We're working hard as a team to improve the course and the infrastructure we have for hosting other courses, and everyone's feedback has been incredibly valuable on our journey thus far :) 

I feel like I have a much better sense of what the current approaches to alignment are, what people are working on and how underdeveloped the field is. In general, it’s been a while since I’ve spent time studying anything so it felt fun just to dedicate time to learning. It also felt empowering to take a field that I’ve heard a lot about at a high level and make it clearer in my mind.

I think doing the Week 0 readings are an easy win for anyone who wants to demystify some of what is going on in ML systems, which I think should be interesting to anyone, even if you’re not interested in alignment.

I became much more motivated to work on making AI go well over the period of the course, I think mainly because it made the problem more concrete but likely just spending more time thinking about it. That said, it’s hard to disentangle this increased motivation from recent events and other factors.

For anyone who is considering the course: TYPE III AUDIO is making audio narrations for the Alignment and Governance courses. The series is due to launch later this month, but some 50+ episodes are already available.

I'm quite glad I took the course! 

Quick takes: 

  1. The main types of value I was getting from the course were: 
    1. Accountability for doing the readings
    2. The chance to use wrong terminology / say things that don't make sense (either when I'm trying to explain something, or when I'm asking a question), and then get corrected (This helped me to develop a more coherent model of what's going on and catch unknown unknowns (at least by transforming them into known unknowns).)
    3. Other resources: links to other readings and explanations
    4. Corrections and clarifications during the sessions
  2. Personal lessons for next time: 
    1. Set aside time to do the readings >24 hours in advance, and send in questions early
      1. (Agree with the facilitator about doing this!)
    2. Use Claude[1] as a personal tutor from the start
      1. This was great. The thing I'd do was, if I was reading something that I was having a hard time following, I'd give Claude some context, then say something like, "I'll now explain this in terms that I understand, or with an analogy or visualization that makes sense to me. Please correct me where I'm misusing terms or saying something wrong." Claude would generally be over-positive and would miss some things, but I'd often get a more technical restatement of what I was trying to say, and this helped me a lot. This was relatively introductory material that wasn't specific to AI safety, so I think Claude was actually performing pretty well. 
    3. Set up a notes and questions doc for myself from the beginning
      1. I now have a messy doc with notes from the different weeks, further readings that we were recommended, and short summaries — I find it quite useful, but the first few weeks are missing because I just hadn't set it up, and was just scribbling down some questions as I read. Having a doc in advance would have been great and would have prompted me to use it.
    4. Actually read the little intro notes for each week on the AGISF website
      1. I think they would have been helpful context for the readings, but I was often skipping them (especially at the beginning), as I didn't really see them as part of the reading. 
    5. I spent a fair amount of time on this and endorse doing that.
      1.  I think the more work I was putting in, the more value I was getting out of the course (there weren't diminishing returns). I was spending probably around 2-5 hours of time before the session each week (there might have been a week when I spent less than 2 hours, and the median time was probably closer to something like 2-3) — in higher-effort weeks, I would explore related writing or referenced texts, and try to get to the point where I could notice the potential weaknesses and confusions I had about the assigned readings. I should flag that I wouldn't always do all the readings very carefully. 
  3. Having some amount of context before starting the course could be useful. If you're not familiar with the overall shape of the argument for existential risk from AI, I imagine that the course structure might be a bit jarring; I moved cohorts a bit in the beginning, but I did feel a bit like it began somewhat abruptly.
  4. Minor points/asides: 
    1. It was just pretty fun.
    2. I have notes on readings I found more and less helpful, and will try to pass those on at some point. 
    3. Facilitators probably matter a lot. 
    4. I enjoyed the readings for some weeks a lot more than I enjoyed them for other weeks. 
    5. I was worried that approaches seemed outdated, in some cases.
    6. I still have pretty broad confusions that I hope to resolve. 
  1. ^

    I was using Claude for dumb reasons — I don't have a strong sense for how it compares to GPT-4 on this. 

I definitely feel it was worth the time for me personally. It was great for learning about the field of AI alignment (problems and proposed solutions). I was hoping the course would spend more time on arguments for and against AI being an x-risk, but unfortunately there was little of that, so it didn't change my mind much.

I think the value I got from the course is that my interactions with AI content are richer. It’s like trying to understand an ecosystem after getting a better prescription for my glasses.

Curated and popular this week
 ·  · 1m read
 · 
 ·  · 5m read
 · 
The AI safety community has grown rapidly since the ChatGPT wake-up call, but available funding doesn’t seem to have kept pace. However, there’s a more recent dynamic that’s created even better funding opportunities, which I witnessed as a recommender in the most recent SFF grant round.[1]   Most philanthropic (vs. government or industry) AI safety funding (>50%) comes from one source: Good Ventures. But they’ve recently stopped funding several categories of work (my own categories, not theirs): * Many Republican-leaning think tanks, such as the Foundation for American Innovation. * “Post-alignment” causes such as digital sentience or regulation of explosive growth. * The rationality community, including LessWrong, Lightcone, SPARC, CFAR, MIRI. * High school outreach, such as Non-trivial. In addition, they are currently not funding (or not fully funding): * Many non-US think tanks, who don’t want to appear influenced by an American organisation (there’s now probably more than 20 of these). * They do fund technical safety non-profits like FAR AI, though they’re probably underfunding this area, in part due to difficulty hiring for this area the last few years (though they’ve hired recently). * Political campaigns, since foundations can’t contribute to them. * Organisations they’ve decided are below their funding bar for whatever reason (e.g. most agent foundations work). OP is not infallible so some of these might still be worth funding. * Nuclear security, since it’s on average less cost-effective than direct AI funding, so isn’t one of the official cause areas (though I wouldn’t be surprised if there were some good opportunities there). This means many of the organisations in these categories have only been able to access a a minority of the available philanthropic capital (in recent history, I’d guess ~25%). In the recent SFF grant round, I estimate they faced a funding bar 1.5 to 3 times higher. This creates a lot of opportunities for other donors
LewisBollard
 ·  · 5m read
 · 
Note: This post was crossposted from the Open Philanthropy Farm Animal Welfare Research Newsletter by the Forum team, with the author's permission. The author may not see or respond to comments on this post. ---------------------------------------- Progress for factory-farmed animals is far too slow. But it is happening. Practices that once seemed permanent — like battery cages and the killing of male chicks — are now on a slow path to extinction. Animals who were once ignored — like fish and even shrimp — are now finally seeing reforms, by the billions. It’s easy to gloss over such numbers. So, as you read the wins below, I encourage you to consider each of these animals as an individual. A hen no longer confined to a cage, a chick no longer macerated alive, a fish no longer dying a prolonged death. I also encourage you to reflect on the role you and your fellow advocates and funders played in these wins. I’m inspired by what you’ve achieved. I hope you will be too. 1. About Cluckin’ Time. Over 1,000 companies globally have now fulfilled their pledges to go cage-free. McDonald’s implemented its pledge in the US and Canada two years ahead of schedule, sparing seven million hens from cages. Subway implemented its pledge in Europe, the Middle East, Oceania, and Indonesia. Yum Brands, owner of KFC and Pizza Hut, reported that for 25,000 of its restaurants it is now 90% cage-free. These are not cheap changes: one UK retailer, Lidl, recently invested £1 billion just to transition part of its egg supply chain to free-range. 2. The Egg-sodus: Cracking Open Cages. In five of Europe’s seven biggest egg markets — France, Germany, Italy, the Netherlands, and the UK — at least two-thirds of hens are now cage-free. In the US, about 40% of hens are — up from a mere 6% a decade ago. In Brazil, where large-scale cage-free production didn’t exist a decade ago, about 15% of hens are now cage-free. And in Japan, where it still barely exists, the nation’s largest egg buyer, Kewpi
Relevant opportunities