What is there in the neural network before learning happens ?
Before training begins, we usually think of a neural network as though nothing has happened yet. The weights are random, model has seen no data and recieved no gradients. We naturally think that whatever intelligence it eventually develops must come later.
But there is something strange about this thought:
Take two untrained networks and give them the same data (and enough capacity to fit the data), same objective to optimize. They will show systematic similarites in the learned function space even when they start from different random initializations. Their optimization trajectories in parameter space might be disjoint but the kinds of functions they learn easily (or resist to learn) will be systematically similar.
This suggests something that is surprisingly easy to overlook:
There is an inherent structure before the first gradient. Before the network knows anything it may already have a characteristic way of learning.
Where does this structure come from and how does it interact with the learning experience ?
In machine learning one might mundanely call it "inductive bias", but giving a phenomenon a familiar name does not resolve the underlying mystery.
This structure isn't inherent to neural networks only. Philosophers argued that a prior structure is what shapes the learning experience for humans. Locke imagined human mind as "white paper", which is initially clean and later furnished through experience [1]. But Hume argued that "finite observations do not determine their own extrapolation" [2]. Experience gives us what has happened, not what it predicts should happen next.
Leibniz beautifully imagines this with veined marble metaphor: before carving the stone into a statue, the structure of the stone makes some forms easier to realize than others [3]. Kant explains this same phenomenon like this: experience alone cannot shape learning unless the mind itself contributes conditions under which experience becomes intelligible [4].
The metaphysics differ but the central intuition is persistent:
If the experience underdetermines what should be learned from it, what does the learner contribute ?
It makes me think that experience alone cannot determine its own interpretation. Learning always takes place through a structure that makes some realizations more natural than others.
And if artificial intelligence is curated on the same principles of human intelligence then it should possess a prior structure to guide the learning process. That's where things get interesting ! Because if it is true then it would mean that a neural network doesn't start neutral and learning becomes a matter of how precisely its prior structure is aligned with the reality it is trying to decode [5].
In that sense, a neutral system (a true blank slate) cannot extrapolate beyond the current experience, it can only memorize it.
The prior structure is not an unfortunate departure from an otherwise neutral learner. It is a part of what makes generalization possible at all.
Now, the interesting question is where does this structure reside in a neural network and is it observable ?
Since the prior is not part of the experience but a guide to learn from it, so it must exist as an inherent characteristic of the learner. Neural Tangent Kernel very intuitively probes this structure [6] through pre-ordained geometry of similarity:
This relation reveals an innate epistemology: a network doesn't know what it will learn yet its architecture has already defined what it means for two experiences to be close, similar or connected. Importantly, it exists before it gets its first glimpse of the world.
One might argue that NTK is a sterile, infinite-width ideal case where the network operates in lazy regime and behaves like a static linear model and fails to capture the richness of true feature learning [7]. But this critique misses the forest for trees. NTK isn't about the learning experience, its about the existence of meaningful structure before learning begins. It tells us that the unformed baby-network is already opinionated.
I now ask myself:
If the structure precedes experience then how does it shape what the experience means ?
If learning inherits a prior structure then the space of possibilities is already tilted. A network might have the capacity of represent a gigantic range of functions but implicitly chose to fit only a subset.
We can see this asymmetry in the spectral bias literature. Neural networks often learn low frequency structure before high frequency structure [8], parameter to function maps can favor simpler functions [9]. Intrinsic dimensions literature makes it even more interesting. Generalizable solutions can be found in surprisingly low dimensional subspaces of the full parameter space [10], while gradient descenet itself can become concentrated in a small set of directions [11].
While the prior might not prescribe a destination but
It guides the learning process by tilting the space of possibilities which makes some directions easier to follow than others.
And that makes sense: if a network treated every possible function or pattern as equally valid, it would be paralyzed by the infinite ways to interpret unseen data.
I think this is one of the reasons heavily overparameterized networks still generalize even though they have the capacity to fit random labels. We can even relate this paradox to Lottery ticket hypothesis [12], it says "training is more like excavation than creation". We do not build massive networks to give them arbitrary capacity but simply to provide enough room to guarantee that a winning prior already exists within the intial randomness.
Capacity describes what is possible; prior structure shapes what is accessible.
Learning never begins from neutrality. Finite evidence does not determine its own extralopation; something about the learner must make some possibilities more accessible than others. And machines inherit this fundamental condition of learning from human intelligence. They must have a prior structure to interpret the experience. A randomly initialized neural network does not know what the world will teach it. Before the first gradient, it already has a way of listening.
[1] Locke, J. (1948). An essay concerning human understanding, 1690.
[2] Hume, D. (1894). An Enquiry concerning the Human Understanding, and an Enquiry concerning the Principles of Morals. Clarendon Press.
[3] Von Leibniz, G. W. F. (1996). Leibniz: New essays on human understanding. Cambridge University Press.
[4] Kant, I. (1908). Critique of pure reason. 1781. Modern Classical Philosophers, Cambridge, MA: Houghton Mifflin, 370-456.
[5] Wolpert, D. H., & Macready, W. G. (1997). No free lunch theorems for optimization. IEEE transactions on evolutionary computation, 1(1), 67-82.
[6] Jacot, A., Gabriel, F., & Hongler, C. (2018). Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31.
[7] Lee, J., Xiao, L., Schoenholz, S., Bahri, Y., Novak, R., Sohl-Dickstein, J., & Pennington, J. (2019). Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32.
[8] Rahaman, N., Baratin, A., Arpit, D., Draxler, F., Lin, M., Hamprecht, F., ... & Courville, A. (2019, May). On the spectral bias of neural networks. In International conference on machine learning (pp. 5301-5310). PMLR.
[9] Valle-Perez, G., Camargo, C. Q., & Louis, A. A. (2018). Deep learning generalizes because the parameter-function map is biased towards simple functions. arXiv preprint arXiv:1805.08522.
[10] Li, C., Farkhoor, H., Liu, R., & Yosinski, J. (2018). Measuring the intrinsic dimension of objective landscapes. arXiv preprint arXiv:1804.08838.
[11] Gur-Ari, G., Roberts, D. A., & Dyer, E. (2018). Gradient descent happens in a tiny subspace. arXiv preprint arXiv:1812.04754.
[12] Frankle, J., & Carbin, M. (2018). The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635.
I wonder if the inductive bias (or the prior structure) changes or evolves when it experiences the world ! Or does it stay rigid invariant throughout training ? If the prior structure really guides learning then how can it stay rigid when the experience itself is evolving ? Finite evidence cannot determine its own extrapolation but the conditions that are necessary for this extrapolation, are those universal ?