This article will argue that the “more is better” relationship with knowledge which is the foundation of science and our modern civilization is simplistic, outdated and increasingly dangerous. Let’s start with a quick analogy which can provide a glimpse of where we’re headed.
Our Evolving Relationship With Food
For most of our history humans have lived near the edge of starvation much of the time. In this scarcity context a “more is better” relationship with food was entirely reasonable.
We live in a new food era now. In our time food is plentiful and readily available in much of the world, and where that’s true more people die of obesity related diseases than die of starvation.
The point here is that a “more is better” relationship with food which was entirely rational for a very long time in an era of food scarcity became outdated and dangerous when transported to a different era characterized by a food explosion. We lucky moderns are required to replace the simplistic “more is better” food paradigm from the earlier era with a more intelligent and sophisticated relationship which can involve complicated cost/benefit calculations.
Our Evolving Relationship With Knowledge
This is where we are in our relationship with knowledge as well. The simplistic “more is better” relationship with knowledge which served us so well for so long now must adapt to meet the challenge of the new environment which it’s success has created.
The modern knowledge explosion obviously brings many benefits, way more than can be listed here, more than our ancestors could have even dreamed of. And although mistakes, missteps and even epic calamities do occur, so far we’ve always managed to clean up the mess, fix the error, learn the lessons, and continue with progress. So what’s the problem??
To understand the threat posed by operating from an outdated relationship with knowledge we need to examine the issue of scale. It is the vast scale of the powers emerging from the knowledge explosion that makes the longstanding progress => mistakes => more progress process that we are used to obsolete.
Erasing The Room For Error
Luckily for the purposes of this article at least, nuclear weapons provide a very easily understood example of how powers of vast scale change the threat landscape by erasing the room for error.
As you know, the nuclear stockpiles of the great powers will have to be managed successfully every single day forever, for as long as those weapons exist. The key thing to note here is that as far as the future of humanity goes, successfully managing such vast power most of the time is no longer sufficient. Doing a pretty good job no longer works. Making a mistake and then fixing it is no longer an option.
In the nuclear era the room for error we’ve always counted on in the past is erased, and one bad day is all it takes to end the possibility for further progress. This is what defines the revolutionary new situation we now find ourselves in, a situation which demands perfection from us.
And Now The Bad News
If nuclear weapons were eliminated entirely the underlying “more is better” knowledge development process which created the nuclear threat would continue to create more vast powers with the potential for crashing civilization.
Each emerging power of vast scale will have to be successfully managed every single day forever because a single mistake with a single such power a single time may be sufficient to crash the system and prevent the opportunity for renewal.
More, Larger, Faster
A key fact of the knowledge explosion is that it feeds back upon itself creating an ever accelerating unfolding of new knowledge, and thus new powers. So not only will emerging powers be larger than what we could produce in the past, and not only will there be more such vast powers than currently, but they will arrive on the scene at an ever faster pace.
Ever more, ever larger powers, delivered at an ever faster pace. Each of these accelerating factors; scale, number, and speed; needs to be graphed against the glacial pace of human maturity development.
Are We Perfect?
There is nothing about thousands of years of human history which suggests that we are capable of the consistently perfect management which powers of vast scale require.
We’ve been able to survive repeated episodes of murderous insanity and other such mistakes in the past only because the powers available to us were limited. As example, we threw conventional explosives at each other with wild abandon in WWII, and were saved from total destruction only because conventional explosives simply aren’t powerful enough to crash civilization.
The Unexamined False Assumption
A simplistic “more is better” relationship with knowledge is built upon the false assumption that human beings will always be able to successfully manage any amount of power which emerges from the knowledge explosion. Simple common sense reveals this assumption to be a wishful thinking fantasy.
We sensibly limit the powers available to kids out of the realistic understanding that their ability to manage power is limited. But then we assume that when children turn 18 they somehow magically acquire the ability to successfully manage any amount of power that the knowledge explosion may deliver.
The irrationality of this assumption is proven beyond doubt by the thousands of hair trigger hydrogen bombs we adults have aimed down our own throats, a stark reality we rarely find interesting enough to comment upon.
Mature? Ready for even more power?
Should We Become Luddites?
Should we turn our backs on knowledge? We don’t have that option. We make our livings on this Earth by knowledge. Knowledge is to humans what wings are to a bird, and fins are to a fish.
To illustrate the path we are now required to walk in our relationship with knowledge, let’s return to the example of food. The solution to obesity is obviously not to stop eating. The solution to obesity is instead to develop a more sophisticated relationship with food, eating what our bodies need, while saying no to excessive consumption.
In the same way, a simplistic “more is better” relationship with knowledge which has served us so well for so long must now make way for a more sophisticated relationship involving complicated cost/benefit calculations. And just as is the case with food, this will sometimes involve saying no to some new knowledge.
Yes, you’re right, it’s true, saying no to any knowledge is typically considered heresy in the age of science. Such reactions are surely understandable, but they are also wishful thinking prisoners of the past.
Nuclear weapons prove that the simplistic “more is better” relationship with knowledge the modern world is built upon is outdated 19th century philosophy which requires updating to meet the existential threats presented by a revolutionary new era.
I was more interested in the obesity analogy and where that might lead, but I think you only ended up doing a less productive recapitulation of Bostrom's vulnerable world hypothesis.
I think "knowledge explosion" might be a more descriptive name for that, I'm not sure it's better strategically (do you really want your theory to be knee-jerk opposed by people who think that you want to generally halt the production or propagation of knowledge?)
Knowledge Obesity though... I'd be interested in talking about that. It's a very good analogy. Look at twitter, it's so easy to digest, extremely presentist, hot takes, conspiracy theories, sounds a lot like the highly processed salted fat and sugar of information consumption, to me.
The places where the analogy breaks are interesting. I suspect that it's going to be very hard to develop standards and consensus about healthy information diets because because modernity relies on specialization, we all have to read very different things. Some people probably should drink from the firehose of digestible news. Most of us shouldn't, but figuring out who should and shouldn't and how they should all fit together is like, the biggest design problem in the world and I've never seen anyone aspire to it. The people who should be doing it, recruiters or knowledge institutions, are all reprehensibly shirking their duty in one way or another.
As MakoYass pointed out this sounds a lot like you are suggesting halting all acquisition of knowledge. While it would handily stop human-created existential risk, I do not think this is possible to implement (as you note, but don't go into how to address). It's sadly another example of solutions like "develop a global culture of coexistence" which would work, but are not practical.
Your post made me think, and I thoroughly applaud the audacity to conceive "unthinkable" directions of inquiry. It made me reconsider my preconceptions! But I think there are some simple reasons this kind of solution isn't more prevalent - with the notable exception of scrupulous work to avoid information hazards. (did you EA is already working to prevent information hazards?)
Counterpoints I would like to see you address:
Its hard to know what we don't know, to avoid discovering it.
Information is extremely useful.
Information can eliminate risks as well as create them.
Historically we have suppressed knowledge incorrectly thinking it would harm us, and this caused great damage.
Other people will discover it, so white hatting is preferable.
If I had to guess, this was downvoted for the wordiness. Your points are buried instead of stated at the top, requiring someone to puzzle through most of your post to reach them.
Another EA thing: your headers are hooks. This is normally good at drawing a reader in, but the EA forum prefers a summary/conclusion of each paragraph to make it easily assessed - if someone agrees with your point already they can skip it and move to the next paragraph. If they disagree they can read the paragraph and be convinced by your evidence.
For the first 8 sections it sounds like you are suggesting 1) great powers create great risks 2) information creates great powers 3) therefore [no additional information acquisition] to stop creating great risks. Its only in your 9th one that you acknowledge that its impossible, but then don't really provide any direction. What do we do given this knowledge? What does this change?
Your conclusions are situated right in between practical folks (who already know knowledge is sometimes bad for society) and theoreticians (who think knowledge is always good; it's the wielding that is the problem) so you might have gotten downvoted by both. It also isn't favorable to the silicon valley pro-progress culture of "move fast and break things" in every area except existential risk. On the other hand, the average person is too suspicious of scientific progress in my opinion, so I want to encourage a pro-science view generally. I'm pretty far on the side of advancing as fast as we can.
Okay, wordiness:
Compare your version with my pared down version:
Obviously I'm hardly one to speak. Just look at how long my comment has gotten. But I think cutting to the meat of your posts will go a long way to making them stronger. I hope this has been helpful feedback.
Unfortunately I have spend too long on this and must go back to spending time reading about biodiversity and ecosystems. :)
To debunk this post, readers might attempt to demonstrate how we human beings, who currently have thousands of massive hydrogen bombs aimed down our own throat, will be able to successfully manage ANY amount of new knowledge and power delivered at ANY rate.
To debunk this post, show us how we human beings, who typically don't find these massive hydrogen bombs interesting enough to discuss, will have the judgement and maturity to consistently make wise choices about how to manage ever more, ever larger powers, delivered at an ever faster pace, without limit.
This is what is required within the "more is better" relationship with knowledge. God like ability. To debunk this post, show us how we become gods.
Thank you.
Show how we "will have the judgement and maturity to consistently make wise choices about how to manage ever more, ever larger powers, delivered at an ever faster pace, without limit."
It's impossible to do anything without mistake, much less forever, much less wisdom, much less humanity. =p We are aware of our own fallibility, so we (humanity and EA) build systems to catch early warning signs and counteract mistakes before they get out of hand.
"Show how we will never make mistakes" is obviously impossible and does not allow for the many other counterpoints I might have within your post. "There is only one way this could be wrong, and its this" is not a very useful discussion for either of us. It suggests you already have your conclusion and will not incorporate input unless it meets your predefined standards. This does not feel like equals working together towards a common goal, but of someone with preconceptions, judging who is allowed to contribute and precluding evidence to the contrary. This may not be your intention, but it is how I read your phrasing.
I find it more useful to note my weakest points, seeking if anyone can find any flaws or better frameworks so I can discover every possible way to improve my worldview and build upon it.