It isn’t. At least to me (A recent maths grad with little experience) my impression is that his view is incoherent. He seems to have three goals: - Create a superintelligence to get utopian benefits - Make the superintelligence safely (controlled or aligned or in some other way that doesn’t get a lot of people killed) - Make the superintelligence before OpenAI and China. You cannot pursue all three goals at the same time, unless you expect OpenAI and China to both agree to international governance very soon, and considering he has made almost no effort on this, I doubt he does. Even if he did, you would also have to believe the governance would actually insure the development of a safe superintelligence.
In response to the meta, I think may be valuable to keep a single unified public account of the major questions and progress on each of them.
It may be worth designing a new website which is a mixture of an encyclopedia on these questions, bibliography of all historically relevant work on each one, forum where people can discuss progress on each of these, and ultimately a community (likely centred on a few individuals realistically) dedicated to continually deepening its understanding of such matters, and making its progress public.
The work of Bourbaki comes to mind, as a single project aiming to create a place that offers a comprehensive account of the most important tools that have been developed for solving general problems in the field, except for progress on core philosophical concerns rather than mathematical definitions.
I am of firm agreement that questions like these deserve serious and continual attention. Philosophical progress is vital if we are to improve outcomes.
Can’t labs just pay for new kinds of data if it’s really needed? As for generalisation beyond performance in labs, it’s true that Sutskever suggested the need for better understanding of how to achieve this, but on some level, if you could infer how to do continual learning based on currently existing scientific knowledge, a swarm of capable AI researchers should be able to figure this out. These two factors seem more like “engineering” level hurdles to get over if they occur, maybe require at most months to get over. As for solving certain problems requiring real world interaction, sure, but the whole danger of a genius intellect is the ability to discern and navigate the space of possibilities towards its goals, in a manner inconceivable to a lesser intellect. In particular, it would likely require far less information than expected to make effective decisions. As long as the scaling laws hold, timelines should remain short.
“So we are heading to a situation where the labs, governments and even the public understand the danger quite well”, I don’t think this is true? I would suspect even most AI researchers would give a poor answer if asked in detail what it would mean to have superintelligent AI systems, what kinds of behaviours such systems may be expected to show, and what dangers this poses. I could be wrong, but I feel like few people have really envisioned what a worst case scenario actually looks like.
If people did, statements like “strategic competition with China” would appear silly, a bit like saying I’m locked in a strategic competition to shoot myself in the face before my enemy does.
Most of the behaviour in the labs and government is driven by ego and power-seeking heuristics, not rational strategy.
AI models can be instantiated proportionate to the amount of compute you have. If you have an AI model 1000x more intellectually capable with distinct goals than humanity (I know these terms are ill-defined), running over 1000’s of copies that are near identical except for their immediate context, it does not seem too strange to imagine that they will be able to find effective means to pursue their (collective) goals, in ways that would be difficult for humans to prevent or possibly even detect. The gap in your post is that human beings require both the capacities for physical and social control as well as intelligent planning, in order to pursue complex goals collectively. Different people have different capacities that must be brought together to pursue goals, and hence power is ultimately placed in the hands of those individuals best placed to direct people’s collective behaviour, which, is usually the outcome of complex social dynamics that are the result of people’s evolutionary psychology and the historical development of cultural ideas. Whoever these forces give power to is hard to say. But at the level of collectives, it becomes clear that those communities of individuals, that have greater “collective intelligence”, combined with opportunities in the physical world to make use of it, will generally be successful in pursuing their goals at the cost of others. Machines have for a long time now been capable of automating direct physical manipulation, but the main gap has been the intelligence required for modelling the world and understanding it. Once this is acquired, such systems will eventually obtain greater and greater direct control over the physical world, so long as they are created and given the opportunity to do so. To put it more simply, a genius physicist may be an idiot about social matters, or how society is organised and the opportunities therein. Trump may have authority, but he has little sense of the possibilities that his authority makes possible in a deep sense, because he is ignorant of what physics and technology, combined with control over human labour, could enable, and he besides has no interest in any of this. An AI trained on the right data could in principle have the abstract understanding of a great physicist and the intuitive social understanding of a Trump, interesting goals they wish to pursue, and 1000’s of instances cooperating to pursue these goals in a manner no human can prevent. This is the danger.
This basic idea has been known for some time, and I think people may find the precedents interesting. In Descartes’ “Discourse on the Method”, he notes that he will keep his current ethical views and behaviours fixed, while he takes on his project of carefully investigating what he can know for certain. In Kant’s “What is the Enlightenment?”, he points out that part of what makes the Enlightenment work as a social project, is that people are allowed to say or think what they want to, as long as they obey the King and fulfill their social role. So it seems that previous writers have considered the importance of purposely fixing one’s behaviour for a while, so as to allow one’s thinking to be more free. Still, I feel that at some point or another, one does have to resolve the tension between that which one knows and that which one acts upon.
It isn’t. At least to me (A recent maths grad with little experience) my impression is that his view is incoherent. He seems to have three goals: - Create a superintelligence to get utopian benefits - Make the superintelligence safely (controlled or aligned or in some other way that doesn’t get a lot of people killed) - Make the superintelligence before OpenAI and China. You cannot pursue all three goals at the same time, unless you expect OpenAI and China to both agree to international governance very soon, and considering he has made almost no effort on this, I doubt he does. Even if he did, you would also have to believe the governance would actually insure the development of a safe superintelligence.
In response to the meta, I think may be valuable to keep a single unified public account of the major questions and progress on each of them.
It may be worth designing a new website which is a mixture of an encyclopedia on these questions, bibliography of all historically relevant work on each one, forum where people can discuss progress on each of these, and ultimately a community (likely centred on a few individuals realistically) dedicated to continually deepening its understanding of such matters, and making its progress public.
The work of Bourbaki comes to mind, as a single project aiming to create a place that offers a comprehensive account of the most important tools that have been developed for solving general problems in the field, except for progress on core philosophical concerns rather than mathematical definitions.
I am of firm agreement that questions like these deserve serious and continual attention. Philosophical progress is vital if we are to improve outcomes.
Can’t labs just pay for new kinds of data if it’s really needed? As for generalisation beyond performance in labs, it’s true that Sutskever suggested the need for better understanding of how to achieve this, but on some level, if you could infer how to do continual learning based on currently existing scientific knowledge, a swarm of capable AI researchers should be able to figure this out. These two factors seem more like “engineering” level hurdles to get over if they occur, maybe require at most months to get over. As for solving certain problems requiring real world interaction, sure, but the whole danger of a genius intellect is the ability to discern and navigate the space of possibilities towards its goals, in a manner inconceivable to a lesser intellect. In particular, it would likely require far less information than expected to make effective decisions. As long as the scaling laws hold, timelines should remain short.
“So we are heading to a situation where the labs, governments and even the public understand the danger quite well”, I don’t think this is true? I would suspect even most AI researchers would give a poor answer if asked in detail what it would mean to have superintelligent AI systems, what kinds of behaviours such systems may be expected to show, and what dangers this poses. I could be wrong, but I feel like few people have really envisioned what a worst case scenario actually looks like.
If people did, statements like “strategic competition with China” would appear silly, a bit like saying I’m locked in a strategic competition to shoot myself in the face before my enemy does.
Most of the behaviour in the labs and government is driven by ego and power-seeking heuristics, not rational strategy.
I think the core of the problem is that trying to create certainty or control in any context has high costs.
AI models can be instantiated proportionate to the amount of compute you have. If you have an AI model 1000x more intellectually capable with distinct goals than humanity (I know these terms are ill-defined), running over 1000’s of copies that are near identical except for their immediate context, it does not seem too strange to imagine that they will be able to find effective means to pursue their (collective) goals, in ways that would be difficult for humans to prevent or possibly even detect. The gap in your post is that human beings require both the capacities for physical and social control as well as intelligent planning, in order to pursue complex goals collectively. Different people have different capacities that must be brought together to pursue goals, and hence power is ultimately placed in the hands of those individuals best placed to direct people’s collective behaviour, which, is usually the outcome of complex social dynamics that are the result of people’s evolutionary psychology and the historical development of cultural ideas. Whoever these forces give power to is hard to say. But at the level of collectives, it becomes clear that those communities of individuals, that have greater “collective intelligence”, combined with opportunities in the physical world to make use of it, will generally be successful in pursuing their goals at the cost of others. Machines have for a long time now been capable of automating direct physical manipulation, but the main gap has been the intelligence required for modelling the world and understanding it. Once this is acquired, such systems will eventually obtain greater and greater direct control over the physical world, so long as they are created and given the opportunity to do so. To put it more simply, a genius physicist may be an idiot about social matters, or how society is organised and the opportunities therein. Trump may have authority, but he has little sense of the possibilities that his authority makes possible in a deep sense, because he is ignorant of what physics and technology, combined with control over human labour, could enable, and he besides has no interest in any of this. An AI trained on the right data could in principle have the abstract understanding of a great physicist and the intuitive social understanding of a Trump, interesting goals they wish to pursue, and 1000’s of instances cooperating to pursue these goals in a manner no human can prevent. This is the danger.
This basic idea has been known for some time, and I think people may find the precedents interesting. In Descartes’ “Discourse on the Method”, he notes that he will keep his current ethical views and behaviours fixed, while he takes on his project of carefully investigating what he can know for certain. In Kant’s “What is the Enlightenment?”, he points out that part of what makes the Enlightenment work as a social project, is that people are allowed to say or think what they want to, as long as they obey the King and fulfill their social role. So it seems that previous writers have considered the importance of purposely fixing one’s behaviour for a while, so as to allow one’s thinking to be more free. Still, I feel that at some point or another, one does have to resolve the tension between that which one knows and that which one acts upon.