Hi everyone! My name is Ram Rachum, and this is my first post here :)
I'm an ex-Google software engineer turned MARL researcher. I want to do MARL research that promotes AI safety. You can read more about my research here and sign up for monthly updates.
I had an idea for a project I could do, and I want you to tell me whether it's been done before.
I want to create a demo of Stuart Russell's "You can't fetch the coffee if you're dead" scenario. I'm imagining a MARL environment where agent 1 can "turn on" agent 2 to prepare coffee for agent 1, and then agent 2 at some point understands how to prevent agent 1 from turning it off again. I'd like to get this behavior to emerge using an RL algorithm like PPO. Crucially, the reward function for agent 2 will be completely innocent.
That way we'll have a video of the "You can't fetch the coffee if you're dead" scenario happening, and we could tweak with that setup to see what kind of changes make it less likely or more likely. We could also show that video to laypeople, and it will likely be much easier for them to connect to such a demo rather than to a verbal description of a thought experiment.
Are there any existing demonstrations of this scenario? Any other insights that you have about this idea would be appreciated.
This seems close to what you're looking for.
Thanks. I looked at their GitHub repo, and it's full of visualizations such as these:
I guess I'll have to dig deeper to understand the meaning of this chart, but right off the bat I'm not seeing a relatable and intuitive demonstration that laypeople can understand. Thanks for sending it anyway, I might find useful things there.
There is a relevant Rob Miles Computerphile video. It does not have a demo component like you are planning, but it does seem to click with laypeople (1M views, top comments generally engaged).