So to be clear, you don't believe in B? And I don't see what mesa-optimers have to do with boxing, if the AI is a box, then so is the mesa-optimiser.
assuming they don't appreciate the existential risk
In the timeline where an actual evil AGI comes about, there would already have been heaps of attacks by buggy AI, killing lots of people and alerting the world to the problem. Active countermeasures can be expected.
So about the box. Is your claim that at
A) at least a few AGI's could argue their way out of a box (ie, if their handlers are easily suggestible/bribeable)
B) Every organisation using an AGI for useful purposes will easily get persuaded to let it out.
To me, A is obviously true, and B is obviously false. But in scenario A, there are multiple AGI's, so things get quite chaotic.
(Also, do you mind explaining more about this "mesa-optimiser"? I don't see how it's relevant to the box...)
I do not think we are at the upper limit of intelligence, nor technology. That was never the point. My point is merely that there are limits to what can be deduced from first principles, no matter how fast you think, or how high ones cognitive abilities are.
This is because there will always be a) assumptions in your reasoning, b) unknown factors and variables, and c) computationally intractable calculations. These are all intertwined with each other.
For example, solving the exact schrodinger equation for a crystal structure requires more compute time than exists in the universe. So you have to come up with approximations and assumptions that reduce the complexity while still allowing useful predictions to be made. The only way to check if these assumptions work is to compare with experimental data. Current methods take several days on a supercomputer to predict the properties of a single defect, and are still only in the right ballpark of the correct answer. It feels very weird to say that an AI could pull off a 3 step 100% perfect murderplan from first principles, while i honestly think it might struggle to model a defect complex with high accuracy.
With that in mind, can you reanswer questions 1 and 2, this time with no internet. Just the man, his memories of a hunter gatherer lifestyle, and a million years to think and ponder.
An entity with access to a nanotech lab who is able to perform experiments in that lab can probably built nanotech, eventually. But that's a much different scenarios to the ones proposed by yudkowsky et al. (the scenario I'm talking about is in point 2)
Can I ask you to give an answer to the following four scenarios? A probability estimate is also fine:
My answers are 1. no, 2. no, 3. no, and 4. almost certainly no.
When I said "absolute perfection", I was not referring to inventing nanotech. I was referring to "protein mixing that makes a nanofactory that makes a 100% effective kill everyone on earth device". Theres a bit of a difference between the two.
Now, when talking about the caveman, I think we've finally arrived at the fundamental disagreement here. As a scientist, and as an empiricist more broadly, I completely reject that the man in the cave could make nanotech.
The number of possible worlds where a cave exists is gargantuan. Theres no way for them to come up with, say, the periodic table, because the majority of elements on there are not accessible with the instruments available within the cave. I can imagine them strolling out with a brilliant plan for nanobots consisting of a complex crystal of byzantium mixed with corillium, only to be informed that neither of those elements exist on earth.
Now, the AI does have more data, but not all data is equally useful. All the cat videos in the world are not gonna get you nanotech (although you might get some of newtonian physics out of it).
Well of course, that's not much of an achievement. A regular human with access to the internet could figure out how to build a nuke, they've already been made!
An AGI trying to build a "protein mixing that makes a nanofactory that makes a 100% effective kill everyone on earth device" is much more analogous to the man locked in a cave.
The immortal man had some information, he can look at the rocks, remember the night sky, etc. He could probably deduce quite a lot, with enough thinking time. But if he wants to get the information required for a nuke, he needs to do scientific experiments that are out of his reach.
The caged AGI has plenty information, and can go very far on existing knowledge. But it's not omniscient. It could probably achieve incredible things, but we're not talking about mere miracles. We're talking about absolute perfection. And that requires testing and empirical evidence. There is not enough computing power in the entire universe to deduce everything from first principles.
Actually that's a fair point, I meant miniscule relative to the amount of speculator money and market caps of these coins. I'll edit the comment.
It's hard to get a good gauge of stablecoins while a sizeable portion of them are possibly fraudulent. For example the largest stablecoin, Tether, is widely considered to be suspect and have been caught out lying about their reserves before. There may be more Terra-Luna style crashes in the making as well (I'm not sure how many algorithmic stablecoins are left, but I doubt a single one will survive long term).
Sending money across borders is a use case for crypto, but I find it highly unlikely to generate enough revenue to pay out much compared to the billions of dollars in speculation. I also doubt this of "alternative" currency use. Stablecoins only work when they are backed by another currency like the USD, so why not just use USD? For digital payments, developing countries have overwhelmingly chosen to use mobile money over crypto, and I trust their judgement on that matter, it seems far more convenient.
Highly agree. The amount of non-speculator money flowing into crypto is miniscule (edit: relative to speculator money). Unless that changes on a very large scale, every dollar made off of crypto is made at somebody else's expense. Which is fine if it's only VC's gambling with each other, but taking out a superbowl ad was a deliberate attempt to sucker in clueless retail investors, knowing full well that most of them would lose their hats.
It's not about the quantity of thinking.
If you locked a prehistoric immortal human in a cave for fifty thousand years, they would not come out with the ability to build a nuke. knowledge and technology require experimentation.