Hi Ghostman ! Is this your real name ? Anyways, you've identified a really fascinating phenomenon. I think your question answers itself. You said, "rigid invariant", invariants do not really change their behavior. Honestly, i don't know if the "prior structure" is really an invariant. Complexity in deep neural networks come from different places like activations, width, loss functions, resnets, optimizers etc. How the prior changes by changing the complexity of the network ? That's an open question. But if we think about it, we'll see that the conditions are not universal per se. With each weight update the space of accessibilities is shrinking. Each weight update finds new paths through the optimization surface. And with each new batch of data the evidence is getting more and more closer to infinite and interpretation is getting deterministic. But i don't know how to join the dots here. That's a really interesting thing you've given me to think about.
Hi Ghostman ! Is this your real name ? Anyways, you've identified a really fascinating phenomenon. I think your question answers itself. You said, "rigid invariant", invariants do not really change their behavior. Honestly, i don't know if the "prior structure" is really an invariant. Complexity in deep neural networks come from different places like activations, width, loss functions, resnets, optimizers etc. How the prior changes by changing the complexity of the network ? That's an open question. But if we think about it, we'll see that the conditions are not universal per se. With each weight update the space of accessibilities is shrinking. Each weight update finds new paths through the optimization surface. And with each new batch of data the evidence is getting more and more closer to infinite and interpretation is getting deterministic. But i don't know how to join the dots here. That's a really interesting thing you've given me to think about.