If you blindly talk about this, it’s plain and don’t provide enough information something like, a system reason when it takes some available information as evidence and then applies some rules and instructions and then a find the best outcome possible for it to take and perform it.
While though it’s accurate in most sense, but it’s not work so well in terms of the quality and how those evaluations are going to come up in a system to perform it and most important at what current stage the system is working on so it can take upon that big step towards achieving a big goal.
The better model could be that a system should understand take the current state and then have observations and evidence and then interpretations and then evaluation and finally the decision which it needs to perform.
The performance and quality of any system is heavily depends on it’s ability to distinguish these area of:
“Where I am now, what does the available evidence tell about this state and, what is the best next transition toward the goal”
The “State” here in AI, is very important, easy to understand and hard to implement in those cases. Because AI doesn’t have things like branching out ideas and long memory interconnections with various data, memory and cross domain knowledge sources like fully able to understand it’s state in which it currently is, and what value and direction it’s carries and what actually needed for it to go forward from there. This is not a problem of memory it’s a problem of having a representation of state.
Too little state → the system becomes short-sighted and loses context.
Too much state → the system becomes overloaded, confused, or computationally expensive.
Because of all the computational complexity running at these AI model, it’s always become a “black box” problem which makes it harder for the creator of those models to actually understand how it’s coming up with new things and finding and concluding a specific information.
Which eventually raises the question of maybe there is no single “current state”. A complex system can simultaneously works on multiple valid states and stages and processes various data at parallels which it does in most cases, it’s matter on whether how we can understand and work on those areas for it’s computational process to produce the answer we want from it and move in the direction we want.
Reasoning is not primary about producing answers. It’s heavily determine about the system current position and selecting the next best possible state or outcome transition.
Just because AI can understand and answer to anything become a parameter of intelligence, it’s more important about it’s ability to choose the right abstraction. It’s sound easy theoretically to just make it able to choose the right and ignore the unwanted things. But making it with that level of capacity and computation it’s become a bottleneck.
For domain specific and intensive research work, the evaluation of goal usually get cluttered into their ability to choose the option and information which is the most general suited not which is important and intensive into that domain research field. It’s matter on how we define the “best” goal for a AI. The best action depend on the current state, constraints and intensity level of knowledge required.
Research requires changing abstraction levels. A broad field question or insight go inside it’s subfields and then find some specific mechanism, use that to find some evidence and then though that found some contractions which can rise to new unresolved question, insights or hypothesis. But a system which always stay at a broad and general session based limit system will produce plausible and shallow research.
A large goal is rarely can be achieved through one decision. It’s can be achieved through thousands of smaller state transition.
The relevance of information for AI systems become a big problem on how it’s work towards achieving it’s goals. Because sometimes one piece of information can be useful in one and cannot in other ones.
It’s become a big meta-selection problem of how it can choose things. In this we’re constantly talking about choosing the goal, state, research, goals, direction. But how?, it can choose those, is it just a programmatical function needed to be done or a structured training and development work needed to done for it to function like that.
If an engineer or a creator of that model create that function “what to choose” then it’s the human who chose the ultimate object not the model finding it for itself. It’ simply RLHF, a reward function, human choose the reward and we give it a goal and we choose the goal.
It’s become a problem of choosing on how can we build a system that discovers what should matter to it from interactions, rather than being explicitly told what should matter.
It’s remain a question of can AI develop it’s own objectives from sufficiently general initial drives and active inference and interacting with goals and decisions which it’s need to goes by. If humans specify what the AI should value, it isn’t finding its ultimate values. If humans specify nothing, then there is no reason for the system to pursue anything at all.
