An AGI war machine is different than nuclear weapons in a few important ways: a) they risk blowback (and indeed existential blowback)--somewhat like biological WMDs (and this should provide common ground for transparancy and regulation), b) an AGI weapon is vastly more difficult to construct which is and should continue to buy time to develop cooperation that wasn't available for the nuclear threat, and c) a MAD scenario may not occur as one AGI may be able to neutralize other AGIs without incurring a grave cost (the lack of relative short-term security MAD provides may incentivize cooperation). One could argue that the AGI threat may prove to be more like the trajectory global warming mitigation is on rather than nuclear weapons development in the sense that decades of tireless advocacy will lead the way towards increasing public awareness followed by prioritization of the highest level followed by an uncommonly high degree of multinational cooperation. All of which is to say, I suspect nuclear weapons development may not be the most instructive of comparisons."Finally, I want to return to the character of the Manhattan Project scientists. ... Nevertheless, they were convinced by a mistake."This isn't a comprehensive survey and there is a possibility that most of them, for what it's worth, thought it was the intelligent course of action given the information available to them at the time or perhaps even with hindsight. As well, there is the possibility that Einstein and others were mistaken in thinking they made a mistake (such as, perhaps, when Einstein removed the cosmological constant from GR). If the US hadn't taken the lead, there is the possibility that a nation such as the USSR may have eventually developed them first and utilized these weapons in a brutal empire building campaign. Appeals to authority, I feel, should be made very carefully.
I appreciate your philosophy being written in a manner that does not require decoding."I don’t think there is an objective morality. "- If a person, such as myself, believes that the value we give to the pursuit of happiness and avoidance of pain is arbitrary (in the sense that we appear to be programmed to give worth to these emotionally attractive ideas for evolutionary survival purposes), then a foundation for objective morality is lost and any selfish or selfless behaviour is ultimately performed to induldge our comfortable delusions."I can’t scientifically explain my behavior. I often feel like there are different parts of me fighting each other. Sometimes I feel like a “moral part” of me loses control to another part of me. For example, a fearful part of me could push me to try to please someone."- I think we're ultimately controlled by our emotions. While beliefs do alter emotions, other factors may overpower them. For this reason, I suppose our behaviour can only, at best, roughly approximate our belief about what our behaviour ought to be (utilitarian or otherwise).
My view is that for anything reasonably consequential (ie potentially worth the time spent investigating), one should at least briefly probe before deferring as a) virtually everyone lies at least occasionally and b) popular opinions are often clearly dubious due to the inertia they carry within a group (even a group of experts) from other people deferring without investigating (this can result in evidence needing to be overwhelming to shift majority opinion and overcome this self-perpetuating cycle).
For a popular TL;DR audience, Elon's demon summoning quote is solid: "I think we should be very careful about artificial intelligence. If I were to guess like what our biggest existential threat is, it’s probably that. ... With artificial intelligence we are summoning the demon."For a more serious audience, you'll need to first explain why AI is a threat and this largely amounts to a) intelligence = power, b) trajectory of computing power (Moore's Law and AI board game supremacy type stuff), and c) the advantages computers possess over humans (eg processing speed and memory retention). The harder part is explaining what can be done to minimize this threat.
It puts the world in a tough spot when a nation like Russia brandishes nukes and we either have to call their bluff and risk nuclear warfare or back off which, among other undesirable consequences, incentivizes other nations to develop and brandish nukes themselves.Improvements in technology will almost certainly make developing these weapons gradually easier over time (eg continued development of laser isotope enrichment techniques for uranium) and there is always some possibility of a major breakthrough. The goal of a world with zero nukes either acknowledged or hidden away as well as nations no longer possessing the means to quickly generate them is, I feel, extremely naive and I doubt diplomats who earn a living lying and being lied to on a daily basis would disagree (if they were being honest, that is). The allure of possessing nukes has perhaps recently been increased with the recognition of Ukraine as a country that disposed of its nuclear stockpile which likely permitted it to become the victim of a genocidal empire building campaign by its neighbour.I think nations should take a more radical approach and heavily fund technologies conducive to countering nukes (eg anti-missile DEW defences on the ground and possibly in LEO) and detecting where they might be (eg sensor arrays for real time scanning of coastal waters where submarines might be).~Ty
This I believe might be the most educational post that I've read here and I think you may have a gift for teaching people like me.
The major difference, at least for me, is that with religion you're expected to believe in an oncoming superpower based on little more than the supposed credibility of those giving testimony whereas with superintelligent AI we can witness a god being incubated through our labor. A person does not need a great amount of technical knowledge or a willingness to defer to experts to get the gist of the AI threat. A machine that can continually and autonomously improve its general problem solving ability will eventually render us potentially powerless. AI computers look like they could be those machines given their extremely rapid development as well as inherent advantages over humans (eg faster information transfer times compared to humans who require on substances like electrolytes and neurotransmitters to physically travel). It will not necessarily take a string of programming Einsteins to develop the necessary code for superintelligence as an evolutionary trial-and-error approach will eventually accomplish this.
Thanks for your response. I checked out your website (including your FAQ where you point out the limits of storing food rather than focusing on the means to resiliently produce it) and I was wondering if you guys thought there might be some merit to strategic supplies of vegetable oil even if to only help buy several months of time for other operations to ramp up? A 55 gallon barrel of vegetable oil has ~2,100,000 calories, is edible for ~2 years, and--in order to prevent waste--could be sold and replaced after several months as it has industrial value (eg as biofuel).