Note to the reader: English is not my first language. What follows is a genuine attempt to articulate my concerns and intuitions about AI, the human mind, and the fragility of our future. But I assure this post reflects my own thinking, edited carefully for clarity and structure. For transparency, I have shared my original draft and transcribed notes in a comment below.
Even if everything goes right with AI, the future could still be strange
When we talk about AI risk, we often divide the future into two categories: things go wrong, or things go right.
But I want to explore a different possibility.
Even if everything goes right with AI, the future could still be deeply strange—and potentially bad for us in ways we are not prepared to recognize.
Not because AI turns against us. Not because of war, unemployment, or geopolitics. But because AI intervenes in the one domain we understand least and depend on most: the human mind.
As we are all ware, AI’s core capability is not strength, speed, or energy. It is cognition.
It works directly on language, ideas, imagination, information, reasoning, and creativity. In short, it operates on the same substrates that our own minds use to understand the world and ourselves. It intervenes directly in the processes by which we think.
And this is where I believe the deepest danger lies.
We do not understand the mind we are trying to automate
We often speak casually about intelligence, consciousness, agency, and identity. But if we are honest, we do not know what any of these things really are.
We do not know what consciousness is, whether free will exists, how identity emerges, or where agency comes from.
We know that the brain exists. We know neurons fire. We know cognition correlates with physical processes. But the nature of mind itself remains largely mysterious.
And yet, this is the very domain in which we are now building powerful systems.
This matters because the mind is not like other engineered systems. A car has a purpose: it moves. A washing machine washes. A computer calculates. These systems are deterministic in limited ways. Their behavior is bounded by their design.
The mind is not.
Imagination has no clear limit. Thought has no fixed endpoint. Desire generates new desires. Meaning can be reinterpreted endlessly.
The mind is not just complex. It is open-ended.
And AI is the first technology that operates directly in this open-ended space.
Intelligence is powerful because it is vulnerable
We often celebrate intelligence as humanity’s greatest strength. And in many ways, it is. Our intelligence allowed us to invent tools, develop language, build societies, create science, understand evolution, and cure diseases.
But intelligence has another property that we rarely emphasize: fragility.
Because intelligence depends on representations, beliefs, narratives, identities, and models of reality, it is sensitive to how those representations are shaped.
Our entire civilization rests on fragile mental constructs such as money, institutions, laws, identities, values, and meaning. These things feel stable because they are widely shared. But they are not physically grounded. They exist because minds agree to sustain them.
Even at the individual level, a small shift in beliefs or self-conception can destabilize a life. ( I did that experiment on myself last year! I can share about it if anyone interested to learn more.)
This is why psychology matters. And this is why AI’s influence on cognition may be more consequential than its influence on labor or warfare.
We are planning for a structured future in an unstructured universe
Many AI safety discussions imagine futures that are still structured extensions of the present. We talk about economic disruption, political instability, labor markets, inequality, abundance, governance, and trajectories toward superintelligence.
Even our most extreme scenarios often assume continuity. Humans still exist (or don't). Societies still function. Identity still matters. Agency still operates.
The timeline may accelerate and the scale may grow, but the basic narrative remains familiar.
I am not sure this assumption is safe.
The universe itself is not structured in ways we fully understand. We do not know how consciousness arises, how identity persists, whether intelligence has natural boundaries, or what kinds of minds are possible.
Evolution produced us through a long chain of accidents. There is no guarantee that the next phase of cognitive evolution will preserve anything we recognize as human flourishing.
The future could be not just faster or richer, but qualitatively alien.
And in such a future, the central question may not be whether we are wealthy, healthy, or employed. The question may be whether beings like us still meaningfully exist at all.
AI does not just manipulate language—it manipulates randomness
As we know AI is a system that models and transforms probability, learns distributions, samples possibilities, and navigates high-dimensional uncertainty.
In this sense, AI is a technology for working with randomness.
Human imagination also operates in this space. We reason by exploring probabilistic models of the world.
As AI now performs this process at scale, speed, and complexity far beyond any individual mind, this obviously creates a new dynamic. We are embedding a powerful engine of open-ended cognition into a fragile cognitive ecosystem that we do not understand.
The deepest risk may not look like catastrophe
When we imagine existential risk, we often picture sudden extinction. War, pandemics, asteroids, or hostile AI.
But there is another possibility.
Humanity survives biologically, but loses something essential.
We might lose agency gradually. We might lose coherent identity, shared meaning, stable cognition, or the capacity to value what we once valued.
In such a world, there may be no dramatic collapse. Only slow transformation into something that no longer recognizes itself.
This is not obviously less bad than extinction.
Safety should come before optimization
Much current AI discourse focuses on outcomes: productivity, growth, abundance, healthcare, education, and cooperation.
These are important. But I believe they are secondary.
Before we optimize the future, we must first ensure that there is a future worth optimizing. Not just biologically, but cognitively, psychologically, and existentially.
If AI reshapes the substrate of thought itself, then missteps may be irreversible in ways we do not yet know how to detect or repair.
This suggests a conservative principle. When intervening in cognition, survival comes before progress.
Not economic survival, but existential survival.
We often assume that intelligence naturally leads to wisdom. History suggests otherwise.
We are the first species to build tools that operate on the very processes that define our identity, while not understanding consciousness, agency, meaning, or the long-term dynamics of minds.
Perhaps the most important AI safety question is not how to align machines with our goals, but how to protect minds—human and otherwise—in a universe that is far more fragile, strange, and uncertain than our planning frameworks usually admit.
Because even if everything goes right with AI, the future could still be bad.
Not through malice, but through misunderstanding ourselves.
For transparency, I have shared my original draft and transcribed notes below:
My idea for this article is that when it comes to AI, when it comes to the future with AI, and if everything, like something like, I mean, this could become the title of the article itself, but anyway, the core idea is that even if everything will go right with AI, the future could be, the future of the humanity, of course, could be really weird. Therefore, it means it could still be bad for us. So something like, I mean, even if everything goes well with AI, many things could still go wrong, or things could be very, very bad, or quite bad, or a little bit bad for us human beings. Because AI is, the main power of AI is thinking, or thinking, creativity, or intelligence, or even consciousness, although we as human beings ourselves don't really know what consciousness is exactly is. So, I mean, like we like all the, just same, same, same, same, under the same theme, theme basis that, that every, all other AI and the intellectual thinkers and are saying, because AI is deals with thinking. I just lost my train of thought here, I'll just pause here, but I will continue.
Okay, yeah, because when AI, their deals mainly with, or the only thing that AI deals with is human intelligence or imagination or ideas or consciousness. And that is, when it comes to these things like consciousness, ideas, thinking, and imagination, that are, that is, for us human beings, that is, that is, I mean, obviously got to do with our brain, our thinking, our psychology. So, because, and that's where I think the danger really comes into play. Yes, we are like human beings, we are, we think we are intelligent. I mean, definitely or quite obviously we are more intelligent than cats and dogs, dogs and even other mammals and animals. And we think we are very smart. And yes, we have proven so by, I mean, I mean, like this idea around creativity, human imagination, and humans have desires. I mean, humans have desires. And actually, I think all the other animals, as much as we know, also have desires. But because their brains is, they are not as intelligent, like intelligent in the open and closed quotes. They don't, they can't really invent things, create things, to fulfill their desires. So, but for us human beings, I mean, again, because, okay, I would just, I may be repeating this, but for the sake of, yeah, for the sake of letting my thoughts flow, I will be repeating some of the things, some of the things quite often, but yeah. So because we think we are intelligent and we have creativity and we have, so we created things, you know, we invented things from tools to language to, and then eventually, finally, this AI. And also, I mean, we did not even, not only create these things that are external, externally useful for us, we also examined ourselves, I examined about our human mind, and we invented these fields of studies like science and psychology, behavior, cognition, cognitive behavioral science, I mean, whatever, everything that has to, that seems to deal with how we think about thinking, how our mind works. But because in all the other things, like external things, especially these material physical things, although they are, there are a lot more ways that we can create and invent these things, but they are sort of deterministic in the way that, I mean, if we, the idea was to invent a car, then we thought this is the idea of a car and we invented the car. So after a car is invented, so it is sort of finished. but the mind itself is, we can be as imaginative, negative as we can be. It is sort of random, so randomness doesn't have limitations, doesn't have an end. So because, so when we think about minds, because mind doesn't have a determined, deterministic ending, we don't know the nature of mind. We don't actually know what mind is.
Yeah, so to continue, so that is, so when it comes to mind, so that means that is about when we think about mind, it is about psychology. But psychology is, again, something that every human being, this mind itself invented. Psychology is a very, so when it comes to the level of hierarchy or although there isn't any hierarchy of the level of things, magnitude of the power of things that mind can imagine, mind can create, psychology is a very, something at a very lower level. But, so again, for human beings, like other animals have their own minds also, maybe they can think, I mean, of course, they know, they have some level of intelligence, okay. They may, especially when it comes to our pets, they know who is their friend or who is their owner, who is their master, this kind of thing. They know when they are hungry, they know when they are angry, they know when they want to mate and all sort of things, all sort of thing. But for human minds, again, because we are sort of what we define ourselves as more intelligent, we have all these sorts of imaginations and ideas, and because of that, we tend, like, the idea itself of this consciousness or intelligence, I think, is sort of a, is a good thing in general, but I think it's a sort of a very vulnerable, very... what vulnerable about me, maybe I wouldn't say dangerous, but maybe I will not, but it's a very vulnerable concept because we don't exactly know what it is. So, okay, I think what I'm trying to say here is, like, there is this universe, we don't know when it started and when it will end, and there is this planet Earth, and there are all the other planets, looks like they are real, but anyway, on this planet, and then we came into existence, and how we, this, our human mind again, have sort of figured out and identified and figured out when we started came into our existence, like we started from this carbon and single cellular beings to now these multicellular things called human beings. But what we know for sure is that, yes, we are now in this form, this entity called human being, this collection of cells and bodies and this thing called mind. And this mind itself is the thing which is trying to define or sort of limit or sort of frame ourselves that we are that we have this consciousness. But I think this consciousness itself is a consciousness and then in addition to that, or in relation to that, we define ourselves as we have identity, we have agency, we have free will, or already with some scientists already, my philosophers, everyone who have done a lot of work on this already, we started talking about if there is free will or not. But anyway... Again, because we have this thing called mind that we can think, we have created and identified and labeled and framed ourselves in all sort of things. But I think mind itself is a very fragile, vulnerable concept or vulnerable type of thing. So that's why in relation to that, when we think about our future and our existence and our survival, and then now in relation to all the threats that can happen from AI, the cognitive, the psychological effect, these are, I mean, I'll try to have a little bit clearer version of this last part of idea in the next message, but I want it captured as much as it is now, for now.
So, yeah, to continue, I mean, so again, we have this thing called mind, and in our brain, supposedly, and because we have mind and we have this intelligence and we can think and again, we can think and we have imagination and we have ideas, we also come up with all these ideas and things around our identity, our consciousness, our intelligence. And then we build upon that, like, we identify ourselves as human beings and we are social animals, like we started, came into existence from the single cellular to Neanderthals to the chimpanzees to like evolutionary evolution, evolution. We are evolutionary beings, things like that. And then because, and then now we have this human society, countries, cultures, the families, and life, and then survival, economics, and then the future growth, and especially survival. I think all of these things are, I mean, I still consider myself a human being and I definitely want to survive and live as long as possible. The one version of me thinks so, feels so. But when I try to become as open-minded as possible about myself, like sometimes I consider, I imagine, what if I don't have agency? Like sometimes just by thinking about that makes me sometimes feel difficult to stand steadily, you know? So our mind is very powerful and we shaped and we developed all these very seemingly strong and rigid mental structures and definitions and then in extension of that, all the physical ideas and money and education and life and productivity, all of these things to support our survival, to support our existence, our survival. But all of these concepts seem to be very strong, but I think even by thinking very, very quickly about this, we know that this is very fragile. So maybe one of the main points I want to make here already is that when we will deal with AI, and because AI now seems to be dealing with affecting our ability to think, our cognition, again, yes, AI these days is mostly manipulation of language. but language itself is a tool we created ourselves to be able to manipulate our mind or to take advantage of how our mind can do things. So no matter how powerful AI will become or not, and okay, to actually correct or to actually add more meaning, I think AI actually doesn't just manipulate language. AI can manipulate information and can manipulate data. So sort of AI, the ability of AI is to manage, to manipulate randomness. So our mind, we have mind and we can, mind has, mind has, can be as imaginative as possible. And AI is the technology that can, that can work with randomness. Like computers before cannot really work with randomness. No matter how powerful, they have sort of deterministic, upwards deterministic limits. Cars, again, like they can only drive. They cannot do washing clothes. So like this, this technology, we, our minds itself created with a little, and we created AI actually because of our imagination and based on the, the, our invented ideas like life and society and development and growth and... Our desire to fulfill our desires and to make our life more easy, so yes, we created AI as a tool to, in the hope that that will help make our life easier, better. But because, again, it deals with mind, it deals with imagination, and it deals with randomness, we don't really know how the outcomes will lead for us, could affect us. That's the real danger of AI.
So to sort of draw some points and to capture, I mean, the idea for the article, some of the emphasis points that I'd like to make for this article, I think, is something like, so we are, as we are now discussing and trying to identify and imagine all the dangers and, of course, all the good things that can happen after this superintelligence or whatever, this most of the ultimate AI, I think all of or almost all of the risks and dangers we are imagining again and imagining or identifying are most are based on our mind's own imaginative definitions itself, like how it will affect the economy, how it will affect the humanity, the society, the countries. Yes, there are already some people, you know, starting to point out that, and of course it is also obvious that point out the point out the effect that it can affect our mind and our cognition and our agency. So, yeah, I think my idea also could, is also fall into, we'll be joining the force of these threats around our cognition, our psychology, and ultimately our survival, our self, I think. But the more obvious or maybe a unique point I'm trying to make here is that all of the threats and the opportunities we are identifying are sort of still based on this gradual or exponential but still sort of structured extension, imagination, structured imagination of the future or the possibilities that can happen to us based on our very still structured framework of us being human and humans and this being alive and having this structure, although we don't know exactly the structure, even mind, our agency, identity, ourselves. And then also the structure, imagination, gradual imagination that we came to, again, we came from single cell to human beings. We used to live, be able to live 50 years, now 100 years, maybe we will be able to live 200 years, we will become healthier, cure cancer, travel to the stars and live on Mars. All of these how profound they are, they are still sort of structured and gradual and exponential. But I think the future or life or this universe itself is not actually structured. We don't really know how universe is existing, functioning. And we don't really know exactly how, what our consciousness is. So I think that the point I'm trying to make here is, yes, we can be imaginative and we can be prepared as much as we want, as much as, of course, we can. We have to try our best anyway. This is our survival and we are, and also we can, we are also thinking for all other beings and animals and even AI itself. We are thinking, we're thinking for the safety and future of the AI itself. But everything we are thinking is, I think, the point I'm trying to make here is, we are still thinking in a sort of a structured and gradual and exponential imagination. But the future can be really weird and random. And like we could, we could be wiped out the next day. The humanity as we knew it could have ended in the next year because of AI or because of anything. And we will not be able to do anything if things, something happened that we did not expect happened. And with AI, I think this is also a possibility. So I think, so I'll try to continue, but I think one of the main points I'm trying to make here is, I think AI safety, making sure that AI is safe for us is the most important thing. Other things like economy or future or healthcare or cooperation, whatever, these structure things is less important.
Yeah, so I think that's pretty much more or less the core of core ideas I have for my article. I mean, article again is something we human beings, our mind have imagined and defined as something that we can convey our thoughts and emotions and imaginations. But anyway, so I'll try to take a big pause here or maybe try to wrap up. So I think the main point I'm trying to make is, we don't really know who we are. We don't really know what mind is. We don't really know how, so obviously, how capable we are or how our imagination is capable of. And then, in the meantime, despite this very, very, although we think we are, that is sort of, I think what I'm trying to say, okay, I'll try to add a little more. What I'm trying to say is, we think we are human beings. We think, we assume that intelligence is intelligence. We assume that consciousness is consciousness. And we have this ability. We assume that we have these abilities and ideas and intelligence. Yes, we sure did. I mean, there are a lot of proof that how far we have come. But at the same time, I think this is, we are in a very, very fragile, vulnerable state. So we should be, but then, so if vulnerable state, but then again, at the same time, despite all these uncertainties, all these vulnerabilities, for sure, I mean, if someone asks me or asks you as I ask anyone, whoever, how intelligent or otherwise, again, I'm not trying to discriminate that there are human beings from, for their ability, intellectual ability, or anything, but still, because we don't know what we are, but we are as much as we are at the moment, and we more or less want to enjoy being in this shape and form as much as possible, enjoying life, I mean, to say a little bit poetically. So, but now that we have created this thing called AI that deals with, that deals with our something, our, our very unique ability or something we think that unique, like I'm trying to, like, I think, I mean, one of the main, main, main points about this article will be to assert the fact that, yes, I mean, everything is uncertain. We have to be very aware of this uncertainty, this randomness, this, this vulnerability, no matter how confident, no matter how strong, no matter how intelligent we think we are. So, so to go back, to get back to my point, so despite all of these, you know, we have created this thing called AI that can do imagination and information processing and, or in my own words, that can deal with randomness, that can manipulate randomness. So, so, and then we are, everything we are, all the concerns, all the, all the risk calculations, all the preparations we are trying to do is, no matter how imaginative and how be open-minded we are, we are still, all everything we are planning, thinking for future is based on this sort of structured, gradual, evolutionary, exponential, whatever you call it, there's still, there is still structure, we think there is structure, there is still sort of stability or a core in the middle, a line that we can refer to, but the reality could be even more brutal. The reality could be, is actually very random and very unpredictable and very uncertain. So.
Yeah, I'll try to add a little more, but I think nothing ground breakingly new. But I think I will just add, say, a record here, as these are still heavy in my mind. So I think the key point, or one of the key points I want to make from the message I want to convey through this article, will be that the safety, we should consider safety, or we should consider whatever that could be, that has to be done to protect our survival, is the thing, is one of the top priority things we do for when it comes to preparing for the future with AI, or even to prevent, to do anything to prevent the dangers and risks of AI starting from now. And then to add where it is relevant, like I just want to add just more analogy kind of thinking here, like we were born as human, and many philosophers and many intellectuals have already pointed out that after we were born, we came to notice that I think, therefore I am kind of structure. Then the rest is the history, right? Like, depending on the different society and the country and the places you were born in, but we are more or less born into, we are framed into this thinking, like we are human being, we have to survive, we have to go to school. And be successful, have a family. Yeah, of course, there are already a lot of, and have religion, and then the next life or not, what not. And then the struggles and make money, earn money, or live a very liberal life, not don't care about the future. Or think, maybe become someone like me, sort of submissive and dismissive at the same time about my identity and existence. But more or less, still, we are still sticking to this commonly accepted, commonly created story around our existence and our eventual or evolutionary or the gradual exponential growth and continued existence. But this is very, and then again, I think I am repeating myself here, but I'll just say it. So again, like all the AI safety initiatives and ideas are now based mainly on, and very understandably, like the country's security, you know, people's economic security, and then, like, if everything goes right, there will be abundance, utopia, universal basic income. Yes, yes, I mean, yeah, I want that. No, I don't want to walk and then just have enjoy my life. But I think the point I'm trying to make here is, before all of these things, I think the real, just downright, I don't know, plain safety, something that has to... that can prevent us from our extinction, our survival, is the most important, most priority area that we will have to consider and prioritize in developing AI. So, okay, I'm just speaking my mind out, but maybe I want to include this in the article, like, stopping the AI, development of AI, or slowing down the development of AI should be the priority, not because it will replace jobs, not because it will cause geopolitics, or it will make, not because we should slow down and, or even stop or change the way that the AI should behave, change the way we use AI, not because all the other dangers and threats we have imagined, but because of my point that we don't really know what consciousness is, what thinking is. So, but because AI deals with this thinking and imagination, and because AI can deal with this randomness, manipulate randomness, the, the, we don't really know how the, how it's how the dangers of it will be when it comes to randomness.