As 2022 comes to an end, I thought it'd be good to maintain a list of "questions that bother me" in thinking about AI safety and alignment. I don't claim I'm the first or only one to have thought about them. I'll keep updating this list.
(The title of this post alludes to the book "Things That Bother Me" by Galen Strawson)
First posted: 12/6/22
Last updated: 1/30/23
General Cognition
- What signs do I need to look for to tell whether a model's cognition has started to emerge, e.g., situational awareness?
- Will a capacity for "doing science" be sufficient condition for general intelligence?
- How easy was it for humans to get science (e.g., compared to evolving to take over the world).
Deception
- What kind of interpretability tools do we need to avoid deception?
- How do we get these interpretability tools and even if we do get them, what if they're like neuroscience for understanding brains (not enough)?
- How can I tell whether a model has found another goal to optimize for during its training?
- What is it that makes a model switch to a goal different from the one set by the designer? How do you prevent it from doing so?
Agent Foundations
- Is the description/modeling of an agent ultimately a mathematical task?
- From where do human agents derive their goals?
- Is value fragile?
Theory of Machine Learning
- What explains the success of deep neural networks?
- Why was connectionism unlikely to succeed?
Epistemology of Alignment (I've written about this here)
- How can we accelerate research?
- Has philosophy ever really helped scientific research e.g., with concept clarification?
- What are some concrete takeaways from the history of science and technology that could be used as advice for alignment researchers and field-builders?
- The emergence of the AI Safety paradigm
Philosophy of Existential Risk
- What is the best way to explain the difference between forecasting extinction scenarios and narratives from chiliasm or escatology?
- What is the best way to think about serious risks in the future without reinforcing a sense of doom?
Teaching and Communication
- Younger people (e.g., my undergraduate students) seem more willing to entertain scenarios of catastrophes and extinction compared to older people (e.g., academics). I find that strange and I don't have a good explanation as to why that is the case.
- The idea of a technological singularity was not difficult to explain and discuss with my students. I think that's surprising given how powerful the weirdness heuristic is.
- The idea of "agency" or "being an agent" was easy to conflate with "consciousness" in philosophical discussions. It's not clear to me why that was the case since I gave a very specific definition of agency.
- Most of my students thought that AI models will never be conscious; it was difficult for them to articulate specific arguments about this, but their intuition seemed to be that there's something uniquely human about consciousness/sentience.
- The "AIs will take our jobs in the future" seems to be a very common concern both among students and academics.
- 80% of a ~25 people classroom thought that philosophy is the right thing to major in if you're interested in how minds work. The question I asked them was: "should you major in philosophy or cognitive science if you want to study how minds work?"
Governance/Strategy
- Should we try to slow down AI progress? What does this mean in concrete steps?
- How should we go about capabilities externalities?
- How should concrete AI risk stories inform/affect AI governance and short-term/long-term future planning?
+1 to sharing lists of questions.
I don't know what 'cognition emerging' means. I suspect the concept is vague/confused.
Why would you want to explain the difference?
I've been asked this question! Or, to be specific, I've been asked something along these lines: human cultures have always been speculating about the end of the world so how is this forecasting x-risk any different?
Some hypotheses to test
- Younger people are more likely to hold and signal radical beliefs and the possibility of extinction is seen as more radical and exciting compared to humanity muddling through like it's done in the past
- Younger people are just beginning to grapple with their own mortality which freaks them out whereas older people are more likely to have made peace with it in some sense
- Older people have survived through many events (including often fairly traumatic ones) so are more likely to have a view of a world that "gets through things" as this aligns with their personal experience
- Older people have been around for a number of past catastrophic predictions that turned out to be wrong?
Nuclear war has been in the news for more than 60 years, and a high priority has been placed on spending those >60 years influencing public opinion on nuclear war via extremely carefully worded statements by spokespeople, which in turn were ghostwritten by spin doctors and other psychological experts with a profoundly strong understanding of news media corporations. This is the main reason, and possibly the only reason, why neither of the two American policial parties or presidential candidates have ever adopted disarmament as part of their nationwide party platform during any elections in that time period.
They weren't successful at their goals 100% of the time (Soviet Propaganda operations may have contributed), but their efforts (and the fact that nuclear war scared people but never happened once for 60+ years) strongly affected the life experiences and cultural development of older people while they were younger.
I would suggest that new paradigms are most likely to establish themselves among the young because they are still in the part of their life where they are figuring out their views.
You should make manifold markets predicting what you’ll think of these questions in a year or 5 years.